Oct 30 14:47:03.291628 ip-10-0-139-204 systemd[1]: Starting Kubernetes Kubelet... Oct 30 14:47:03.691953 ip-10-0-139-204 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:47:03.691953 ip-10-0-139-204 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 14:47:03.691953 ip-10-0-139-204 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:47:03.691953 ip-10-0-139-204 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:47:03.696042 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.695982 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.696042 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696041 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.696042 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696045 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696048 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696051 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696054 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696056 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696059 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696062 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696065 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696068 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696071 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696073 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696077 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696079 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696082 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696084 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696087 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696089 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696092 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696095 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696097 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.696136 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696100 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696102 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696105 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696115 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696118 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696121 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696123 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696126 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696129 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696132 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696134 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696137 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696140 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696143 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696146 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696149 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696153 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696158 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696161 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.696599 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696164 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696166 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696169 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696171 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696174 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696177 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696180 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696182 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696185 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696187 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696190 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696193 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696195 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696198 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696202 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696207 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696210 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696213 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696216 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.697044 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696218 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696221 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696224 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696226 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696229 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696232 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696235 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696238 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696240 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696243 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696245 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696248 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696250 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696253 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696256 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696258 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696261 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696263 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696266 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696269 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.697519 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696272 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696275 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696277 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696280 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696282 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696285 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696287 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696290 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696293 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696296 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696304 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696307 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696309 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696312 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696315 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696317 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696320 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696323 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696326 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.698015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696329 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696332 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696334 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696336 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696339 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696342 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.696346 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698140 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698147 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698150 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698153 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698156 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698158 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698161 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698164 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698167 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698169 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698172 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698174 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698177 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.698466 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698180 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698183 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698186 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698188 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698190 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698193 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698196 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698198 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698200 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698203 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698205 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698208 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698211 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698214 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698217 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698220 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698222 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698225 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698228 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698230 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.698971 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698232 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698235 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698237 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698240 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698243 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698245 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698248 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698251 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698253 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698256 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698258 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698261 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698264 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698266 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698269 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698271 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698274 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698276 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698279 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.699448 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698283 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698287 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698290 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698292 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698295 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698297 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698302 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698306 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698309 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698312 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698315 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698317 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698320 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698323 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698326 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698328 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698330 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698333 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698336 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.699931 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698338 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698341 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698344 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698363 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698365 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698368 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698371 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698375 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698378 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698381 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698384 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698386 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698389 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698391 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698394 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698396 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698399 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698402 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698404 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698409 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.700390 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698413 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698415 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698418 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698421 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698423 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698426 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698429 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698431 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698433 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698436 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698438 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698441 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698443 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698446 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698449 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698918 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698924 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698927 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698930 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698934 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.700936 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698936 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698939 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698942 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698945 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698948 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698950 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698953 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698955 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698958 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698960 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698963 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698968 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698971 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698974 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698977 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698980 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698982 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698985 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698988 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.701429 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698990 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698993 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698995 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.698998 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699000 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699003 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699005 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699008 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699017 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699020 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699022 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699025 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699027 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699030 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699033 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699035 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699038 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699041 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699044 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699047 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.701889 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699051 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699054 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699057 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699060 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699063 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699066 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699068 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699071 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699073 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699076 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699078 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699081 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699083 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699085 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699088 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699091 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699093 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699095 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699098 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.702362 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699100 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699103 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699106 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699108 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699111 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699113 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699116 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699118 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699121 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699123 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699126 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699129 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699131 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699134 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699136 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699139 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699141 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699145 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699147 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699149 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.702798 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699152 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699155 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699157 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699160 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699162 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699165 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699167 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699170 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699172 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699175 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699178 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699181 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699184 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699187 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699190 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699193 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699195 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699198 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699201 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.703265 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699204 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699206 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699209 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699212 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699304 2571 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699312 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699320 2571 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699327 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699333 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699338 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699359 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699364 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699367 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699370 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699374 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699377 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699380 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699383 2571 flags.go:64] FLAG: --cgroup-root="" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699386 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699389 2571 flags.go:64] FLAG: --client-ca-file="" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699392 2571 flags.go:64] FLAG: --cloud-provider="external" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699395 2571 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699400 2571 flags.go:64] FLAG: --cluster-domain="" Oct 30 14:47:03.703712 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699403 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699406 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699410 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699413 2571 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699417 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699420 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699424 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699427 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699430 2571 flags.go:64] FLAG: --contention-profiling="false" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699433 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699436 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699439 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699442 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699447 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699450 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699453 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699456 2571 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699460 2571 flags.go:64] FLAG: --enable-server="true" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699463 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699467 2571 flags.go:64] FLAG: --event-burst="100" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699470 2571 flags.go:64] FLAG: --event-qps="50" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699473 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699476 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699479 2571 flags.go:64] FLAG: --eviction-hard="" Oct 30 14:47:03.704236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699483 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699486 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699489 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699492 2571 flags.go:64] FLAG: --eviction-soft="" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699495 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699498 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699501 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699504 2571 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699507 2571 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699510 2571 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699512 2571 flags.go:64] FLAG: --feature-gates="" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699516 2571 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699519 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699522 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699526 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699529 2571 flags.go:64] FLAG: --healthz-port="10248" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699532 2571 flags.go:64] FLAG: --help="false" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699535 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-139-204.ec2.internal" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699538 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699544 2571 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699547 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699550 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699553 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699556 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 14:47:03.704862 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699560 2571 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699563 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699566 2571 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699569 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699572 2571 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699575 2571 flags.go:64] FLAG: --kube-reserved="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699578 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699581 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699585 2571 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699588 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699591 2571 flags.go:64] FLAG: --lock-file="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699593 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699596 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699599 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699605 2571 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699608 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699611 2571 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699614 2571 flags.go:64] FLAG: --logging-format="text" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699617 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699620 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699623 2571 flags.go:64] FLAG: --manifest-url="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699626 2571 flags.go:64] FLAG: --manifest-url-header="" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699631 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699634 2571 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699638 2571 flags.go:64] FLAG: --max-pods="110" Oct 30 14:47:03.705449 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699642 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699644 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699649 2571 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699668 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699673 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699676 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699680 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699688 2571 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699691 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699694 2571 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699697 2571 flags.go:64] FLAG: --pod-cidr="" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699700 2571 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699703 2571 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699706 2571 flags.go:64] FLAG: --pods-per-core="0" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699709 2571 flags.go:64] FLAG: --port="10250" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699712 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699715 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05440f16bb9cda115" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699719 2571 flags.go:64] FLAG: --qos-reserved="" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699721 2571 flags.go:64] FLAG: --read-only-port="10255" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699725 2571 flags.go:64] FLAG: --register-node="true" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699727 2571 flags.go:64] FLAG: --register-with-taints="" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699731 2571 flags.go:64] FLAG: --registry-burst="10" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699734 2571 flags.go:64] FLAG: --registry-qps="5" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699737 2571 flags.go:64] FLAG: --reserved-cpus="" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699740 2571 flags.go:64] FLAG: --reserved-memory="" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699744 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 14:47:03.706068 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699754 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699757 2571 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699760 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699763 2571 flags.go:64] FLAG: --runonce="false" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699766 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699770 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699773 2571 flags.go:64] FLAG: --seccomp-default="false" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699776 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699781 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699784 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699787 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699790 2571 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699793 2571 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699797 2571 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699800 2571 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699803 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699807 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699810 2571 flags.go:64] FLAG: --system-cgroups="" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699813 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699818 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699821 2571 flags.go:64] FLAG: --tls-cert-file="" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699824 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699828 2571 flags.go:64] FLAG: --tls-min-version="" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699831 2571 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699834 2571 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 14:47:03.706701 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699837 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699840 2571 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699843 2571 flags.go:64] FLAG: --v="2" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699847 2571 flags.go:64] FLAG: --version="false" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699851 2571 flags.go:64] FLAG: --vmodule="" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699855 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.699858 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699988 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699992 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699995 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.699998 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700001 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700004 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700007 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700010 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700014 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700017 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700020 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700022 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700025 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700030 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.707275 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700034 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700038 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700041 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700044 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700046 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700049 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700052 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700055 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700057 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700060 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700063 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700065 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700068 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700071 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700073 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700076 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700078 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700081 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700085 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700088 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.707784 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700091 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700093 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700096 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700099 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700101 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700104 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700108 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700110 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700113 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700116 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700118 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700122 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700125 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700127 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700130 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700133 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700135 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700138 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700141 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700143 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.708256 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700146 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700148 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700151 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700153 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700156 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700159 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700161 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700164 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700166 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700169 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700171 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700174 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700176 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700179 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700182 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700184 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700187 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700189 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700195 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.708778 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700198 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700201 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700204 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700206 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700210 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700213 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700215 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700218 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700220 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700223 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700226 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700228 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700231 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700233 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700236 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700239 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700241 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700244 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700246 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.709239 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700249 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700252 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700254 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700257 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700259 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700262 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700264 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700267 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700270 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700272 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700275 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700278 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700282 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700284 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.700311 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700344 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.709705 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700361 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700365 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700369 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700372 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700375 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700377 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700380 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700383 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700385 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700388 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700390 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700393 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700396 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700399 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700401 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700404 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700407 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700409 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700412 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700414 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.710091 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700417 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700419 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700422 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700425 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700427 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700430 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700432 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700435 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700438 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700441 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700443 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700446 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700449 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700451 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700455 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700458 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700460 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700463 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700466 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.710601 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700468 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700471 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700473 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700477 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700479 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700482 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700485 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700487 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700490 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700492 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700495 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700497 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700500 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700502 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700505 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700507 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700510 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700512 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700515 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.711057 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700517 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700520 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700524 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700527 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700529 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700532 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700534 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700537 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700542 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700544 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700547 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700550 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700562 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700565 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700568 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700570 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700573 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700575 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700578 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.711508 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700580 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700583 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700586 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700588 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700594 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700597 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700600 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700603 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700605 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700608 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700611 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700615 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700619 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700621 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700624 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700627 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700630 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700633 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700635 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700638 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.711948 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700643 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700648 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700651 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700654 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700656 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700659 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700662 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.700664 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.707306 2571 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.707317 2571 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.709035 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.709042 2571 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709096 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709102 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709105 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709109 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709111 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709114 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709117 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709119 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.712428 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709122 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709125 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709128 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709131 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709133 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709136 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709139 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709142 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709145 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709147 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709150 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709153 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709156 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709159 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709162 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709165 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709167 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709170 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709172 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.712921 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709175 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709178 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709181 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709183 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709186 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709189 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709191 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709194 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709196 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709199 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709202 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709204 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709207 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709209 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709212 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709214 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709217 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709219 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709222 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709224 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.713441 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709227 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709230 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709232 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709235 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709238 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709241 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709244 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709247 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709250 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709252 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709255 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709258 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709260 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709263 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709265 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709268 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709270 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709274 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709278 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.713945 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709281 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709283 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709286 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709288 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709291 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709294 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709296 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709299 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709302 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709304 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709307 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709310 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709312 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709315 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709318 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709320 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709324 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709328 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.714461 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709331 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709333 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709338 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709340 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709343 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709357 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709361 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709363 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709366 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709368 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709371 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709374 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709377 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709379 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709382 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709384 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709387 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709389 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709392 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709395 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.714877 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709397 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709400 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.709426 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709572 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709576 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709579 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709583 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709586 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709589 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709592 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709594 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709597 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709600 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709602 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709605 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709608 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.715335 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709611 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709613 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709616 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709618 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709620 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709623 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709625 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709629 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709633 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709636 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709639 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709641 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709643 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709646 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709648 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709651 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709653 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709656 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709658 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.715730 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709661 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709663 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709665 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709668 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709671 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709673 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709676 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709678 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709681 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709683 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709686 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709689 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709691 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709694 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709697 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709699 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709702 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709704 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709707 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709710 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.716178 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709712 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709715 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709718 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709720 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709723 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709725 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709728 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709730 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709733 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709735 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709738 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709741 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709743 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709746 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709748 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709750 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709753 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709756 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709759 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.716639 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709761 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709763 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709766 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709769 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709771 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709774 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709776 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709779 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709782 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709784 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709787 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709789 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709792 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709794 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709797 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709799 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709802 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709804 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709806 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.717083 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709809 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709811 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709814 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709816 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709819 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709821 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709823 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709826 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709828 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709831 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709833 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709836 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709838 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709842 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709845 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709848 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.717674 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.709870 2571 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709900 2571 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709903 2571 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709906 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709908 2571 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709911 2571 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709914 2571 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709917 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709919 2571 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709921 2571 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709924 2571 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709926 2571 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709929 2571 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709931 2571 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709934 2571 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709936 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709939 2571 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709943 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709946 2571 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709948 2571 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:47:03.718066 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709950 2571 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709953 2571 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709956 2571 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709958 2571 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709961 2571 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709963 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709966 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709968 2571 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709971 2571 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709973 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709976 2571 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709980 2571 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709982 2571 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709985 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709988 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709990 2571 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709992 2571 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709995 2571 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.709998 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710001 2571 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:47:03.718548 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710004 2571 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710006 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710009 2571 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710011 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710014 2571 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710016 2571 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710019 2571 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710021 2571 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710024 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710028 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710030 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710033 2571 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710035 2571 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710038 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710040 2571 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710043 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710045 2571 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710048 2571 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710051 2571 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:47:03.719013 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710053 2571 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710056 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710059 2571 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710061 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710065 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710067 2571 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710070 2571 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710073 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710075 2571 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710078 2571 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710080 2571 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710083 2571 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710086 2571 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710088 2571 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710091 2571 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710093 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710096 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710099 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710102 2571 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:47:03.719604 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710104 2571 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710107 2571 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710109 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710113 2571 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710117 2571 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710120 2571 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710123 2571 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710125 2571 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710128 2571 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710130 2571 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710133 2571 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710136 2571 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710138 2571 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710141 2571 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710143 2571 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710146 2571 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710149 2571 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710152 2571 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710155 2571 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710157 2571 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:47:03.720240 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710160 2571 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710162 2571 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710165 2571 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710167 2571 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710169 2571 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710172 2571 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710175 2571 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710180 2571 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:03.710183 2571 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.710231 2571 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.712557 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.713629 2571 server.go:1014] "Starting client certificate rotation" Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.713728 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.713766 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.715914 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:47:03.721015 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.717291 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:47:03.729790 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.729773 2571 log.go:25] "Validated CRI v1 runtime API" Oct 30 14:47:03.733545 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.733533 2571 log.go:25] "Validated CRI v1 image API" Oct 30 14:47:03.734318 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.734297 2571 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 14:47:03.740529 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.740505 2571 fs.go:135] Filesystem UUIDs: map[65ed9b72-b0a0-483d-849e-70ebf51a4b8e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9acda3fc-1fa8-49c2-b729-64eac504437c:/dev/nvme0n1p4] Oct 30 14:47:03.740599 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.740528 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 14:47:03.746622 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.746512 2571 manager.go:217] Machine: {Timestamp:2026-10-30 14:47:03.744022986 +0000 UTC m=+0.362552890 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100513 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20fe6dbad8527ab1d85b8978ee61ad SystemUUID:ec20fe6d-bad8-527a-b1d8-5b8978ee61ad BootID:1aceed49-55ae-47f4-85c9-74906c8da00c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:4a:1c:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:4a:1c:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:74:a6:70:97:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 14:47:03.747222 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.747212 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 14:47:03.747294 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.747283 2571 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 14:47:03.750981 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.750956 2571 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 14:47:03.751114 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.750983 2571 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 14:47:03.751873 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.751862 2571 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 14:47:03.751911 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.751876 2571 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 14:47:03.751911 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.751889 2571 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:47:03.751911 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.751898 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:47:03.753215 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.753205 2571 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 14:47:03.758974 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.758963 2571 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 14:47:03.759090 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.759081 2571 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 14:47:03.759954 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.759944 2571 kubelet.go:493] "Attempting to sync node with API server" Oct 30 14:47:03.759998 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.759958 2571 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 14:47:03.759998 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.759970 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 14:47:03.759998 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.759978 2571 kubelet.go:405] "Adding apiserver pod source" Oct 30 14:47:03.759998 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.759986 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 14:47:03.760262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.760251 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:47:03.762705 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.762686 2571 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 14:47:03.769132 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.769117 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:47:03.770121 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770109 2571 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 14:47:03.770168 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770139 2571 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 14:47:03.770316 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770302 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 14:47:03.770372 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770332 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 14:47:03.770372 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770358 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 14:47:03.770433 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770373 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 14:47:03.770433 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770389 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 14:47:03.770433 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.770397 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 14:47:03.771044 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.771034 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 14:47:03.771077 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.771044 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 14:47:03.771077 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.771055 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 14:47:03.771077 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.771064 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 14:47:03.771077 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.771073 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 14:47:03.771280 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.771082 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 14:47:03.772880 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.772869 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 14:47:03.772880 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.772881 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 14:47:03.776641 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.776628 2571 server.go:1263] "Started kubelet" Oct 30 14:47:03.776786 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.776740 2571 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 14:47:03.777065 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.776975 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 14:47:03.777132 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.777121 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 14:47:03.777319 ip-10-0-139-204 systemd[1]: Started Kubernetes Kubelet. Oct 30 14:47:03.779414 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.778722 2571 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 14:47:03.784024 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.784007 2571 server.go:317] "Adding debug handlers to kubelet server" Oct 30 14:47:03.786113 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.786092 2571 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 14:47:03.789798 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.789783 2571 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 14:47:03.789798 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.789794 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 14:47:03.789928 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.789881 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:03.790608 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790586 2571 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 14:47:03.790608 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790604 2571 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 14:47:03.790751 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790590 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 14:47:03.790751 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790747 2571 reconstruct.go:97] "Volume reconstruction finished" Oct 30 14:47:03.790841 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790757 2571 reconciler.go:29] "Reconciler: start to sync state" Oct 30 14:47:03.790841 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790813 2571 factory.go:55] Registering systemd factory Oct 30 14:47:03.790841 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.790826 2571 factory.go:223] Registration of the systemd container factory successfully Oct 30 14:47:03.791103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.791090 2571 factory.go:153] Registering CRI-O factory Oct 30 14:47:03.791103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.791103 2571 factory.go:223] Registration of the crio container factory successfully Oct 30 14:47:03.791220 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.791146 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 14:47:03.791220 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.791165 2571 factory.go:103] Registering Raw factory Oct 30 14:47:03.791220 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.791176 2571 manager.go:1199] Started watching for new ooms in manager Oct 30 14:47:03.792256 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.792240 2571 manager.go:319] Starting recovery of all containers Oct 30 14:47:03.801782 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.801763 2571 manager.go:324] Recovery completed Oct 30 14:47:03.806439 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.806415 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 14:47:03.807218 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.807203 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:03.808869 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.808851 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:03.808947 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.808879 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:03.808947 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.808892 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:03.809236 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.809209 2571 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 14:47:03.809236 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.809219 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-204.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 14:47:03.809387 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.809370 2571 cpu_manager.go:226] "Starting" policy="none" Oct 30 14:47:03.809387 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.809384 2571 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 14:47:03.809495 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.809404 2571 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 14:47:03.809495 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.809477 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:03.810116 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.810099 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:03.810192 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.810123 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:03.810192 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.810133 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:03.810469 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.809226 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd797811a3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.776588195 +0000 UTC m=+0.395118100,LastTimestamp:2026-10-30 14:47:03.776588195 +0000 UTC m=+0.395118100,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.814128 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.814113 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v752b" Oct 30 14:47:03.826244 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.826225 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-v752b" Oct 30 14:47:03.836872 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.836808 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd7b649b91 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-204.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.808867217 +0000 UTC m=+0.427397129,LastTimestamp:2026-10-30 14:47:03.808867217 +0000 UTC m=+0.427397129,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.837576 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.837547 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 14:47:03.837576 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.837552 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 14:47:03.849660 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.849639 2571 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-204.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 14:47:03.856887 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.856832 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd7b64e26c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-204.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.808885356 +0000 UTC m=+0.427415262,LastTimestamp:2026-10-30 14:47:03.808885356 +0000 UTC m=+0.427415262,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.886287 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd7b651373 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-204.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.808897907 +0000 UTC m=+0.427427815,LastTimestamp:2026-10-30 14:47:03.808897907 +0000 UTC m=+0.427427815,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.890185 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.905955 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-204.ec2.internal.18e355fd7b649b91\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd7b649b91 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-204.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.808867217 +0000 UTC m=+0.427397129,LastTimestamp:2026-10-30 14:47:03.81011246 +0000 UTC m=+0.428642365,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.928846 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-204.ec2.internal.18e355fd7b64e26c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd7b64e26c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-204.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.808885356 +0000 UTC m=+0.427415262,LastTimestamp:2026-10-30 14:47:03.810127678 +0000 UTC m=+0.428657583,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.931412 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.932497 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.932957 2571 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.932990 2571 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.933029 2571 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.944874 2571 policy_none.go:50] "Start" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.944888 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 14:47:03.946262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.944898 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 14:47:03.948527 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.946990 2571 policy_none.go:44] "Start" Oct 30 14:47:03.951963 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.951908 2571 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-204.ec2.internal.18e355fd7b651373\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-204.ec2.internal.18e355fd7b651373 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-204.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-204.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-204.ec2.internal,},FirstTimestamp:2026-10-30 14:47:03.808897907 +0000 UTC m=+0.427427815,LastTimestamp:2026-10-30 14:47:03.810136625 +0000 UTC m=+0.428666530,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-204.ec2.internal,}" Oct 30 14:47:03.978485 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.978466 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:03.985875 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.985842 2571 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 14:47:03.988013 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.987995 2571 manager.go:341] "Starting Device Plugin manager" Oct 30 14:47:03.988574 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.988559 2571 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 14:47:03.988574 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.988574 2571 server.go:87] "Starting device plugin registration server" Oct 30 14:47:03.988751 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.988741 2571 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 14:47:03.988787 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.988752 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 14:47:03.988895 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.988868 2571 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 14:47:03.989027 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.988970 2571 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 14:47:03.989027 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:03.988982 2571 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 14:47:03.989321 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.989304 2571 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 14:47:03.989396 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:03.989358 2571 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.033389 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.033364 2571 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"] Oct 30 14:47:04.033480 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.033419 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:04.034179 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.034165 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:04.034270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.034189 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:04.034270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.034199 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:04.036475 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.036462 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:04.036630 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.036616 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.036678 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.036643 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:04.037183 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.037168 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:04.037270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.037184 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:04.037270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.037206 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:04.037270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.037215 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:04.037270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.037192 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:04.037438 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.037278 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:04.037765 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.037749 2571 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-204.ec2.internal\" not found" node="ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.039439 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.039425 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.039487 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.039450 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:04.040100 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.040083 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:04.040181 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.040111 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:04.040181 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.040127 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:04.065969 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.065949 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-204.ec2.internal\" not found" node="ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.068300 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.068282 2571 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-204.ec2.internal\" not found" node="ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.089171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.089132 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:04.089759 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.089743 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:04.089836 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.089769 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:04.089836 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.089779 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:04.089836 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.089796 2571 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.091952 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.091936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.092001 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.091959 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.092001 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.091981 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c47c2ff730e7510466e8747f4e5f45ca-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"c47c2ff730e7510466e8747f4e5f45ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.108820 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.108803 2571 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.108875 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.108820 2571 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-204.ec2.internal\": node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.147997 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.147981 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.192648 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.192630 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.192718 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.192654 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.192718 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.192669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c47c2ff730e7510466e8747f4e5f45ca-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"c47c2ff730e7510466e8747f4e5f45ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.192718 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.192712 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c47c2ff730e7510466e8747f4e5f45ca-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"c47c2ff730e7510466e8747f4e5f45ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.192820 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.192732 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.192820 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.192753 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.248609 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.248558 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.348810 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.348787 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.369257 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.369238 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.371433 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.371418 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:47:04.449139 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.449015 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.549598 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.549541 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.649982 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.649953 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.713515 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.713490 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 14:47:04.713873 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.713610 2571 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 14:47:04.750776 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.750754 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.851777 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.851721 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.872329 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:04.872300 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47c2ff730e7510466e8747f4e5f45ca.slice/crio-b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c WatchSource:0}: Error finding container b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c: Status 404 returned error can't find the container with id b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c Oct 30 14:47:04.877511 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.877495 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:47:04.896307 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:04.896287 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8f5d16b96687b2a915ddaffb597231.slice/crio-138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66 WatchSource:0}: Error finding container 138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66: Status 404 returned error can't find the container with id 138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66 Oct 30 14:47:04.935671 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.935628 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"4c8f5d16b96687b2a915ddaffb597231","Type":"ContainerStarted","Data":"138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66"} Oct 30 14:47:04.936630 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.936610 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" event={"ID":"c47c2ff730e7510466e8747f4e5f45ca","Type":"ContainerStarted","Data":"b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c"} Oct 30 14:47:04.945933 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.945910 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:42:04 +0000 UTC" deadline="2028-05-25 21:08:33.969462088 +0000 UTC" Oct 30 14:47:04.945933 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.945931 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13758h21m29.02353391s" Oct 30 14:47:04.952065 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:04.952047 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:04.961224 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:04.961212 2571 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:05.010722 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.010705 2571 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:47:05.011559 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.011545 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:47:05.011634 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.011572 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:47:05.011634 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.011586 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:47:05.052346 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:05.052319 2571 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:47:05.116721 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.116676 2571 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:05.190606 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.190585 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:47:05.200490 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.200467 2571 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:05.213856 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.213837 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:47:05.214680 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.214667 2571 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:47:05.240138 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.240113 2571 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:47:05.715802 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.715771 2571 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:47:05.761024 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.760995 2571 apiserver.go:52] "Watching apiserver" Oct 30 14:47:05.790616 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.790583 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 14:47:05.794238 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.794213 2571 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 14:47:05.794647 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.794626 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bz4fj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2","openshift-multus/network-metrics-daemon-cjwh8","openshift-dns/node-resolver-q5b49","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal","openshift-network-operator/iptables-alerter-hzqxz","kube-system/konnectivity-agent-sbcnp","kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal","openshift-multus/multus-additional-cni-plugins-rgsm9","openshift-multus/multus-hjchw","openshift-cluster-node-tuning-operator/tuned-dmr9v","openshift-image-registry/node-ca-vvpcd","openshift-network-diagnostics/network-check-target-fnbqw","openshift-ovn-kubernetes/ovnkube-node-wsrm2"] Oct 30 14:47:05.797478 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.797455 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:05.799813 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.799787 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.802727 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.802446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:05.802727 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:05.802537 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:05.805264 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.804932 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:05.808204 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.807553 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.810210 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.809915 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:05.813391 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.812745 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.814332 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.814314 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 14:47:05.814528 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.814494 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8r54d\"" Oct 30 14:47:05.814635 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.814548 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 14:47:05.815920 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.815885 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:05.816029 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:05.815956 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:05.816174 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.816145 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:47:05.816271 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.816185 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw2pz\"" Oct 30 14:47:05.816271 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.816261 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.818700 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.818681 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.821505 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.821460 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.821730 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.821710 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 14:47:05.821963 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.821878 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.821963 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.821930 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g6822\"" Oct 30 14:47:05.823804 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.823783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:05.823898 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:05.823858 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:05.826212 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.826194 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:05.859742 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.859598 2571 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 14:47:05.859742 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.859669 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4gc4r" Oct 30 14:47:05.876093 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.876065 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4gc4r" Oct 30 14:47:05.892188 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.892166 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 14:47:05.903760 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903729 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-etc-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.903861 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903762 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cni-binary-copy\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.903861 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903787 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-cnibin\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.903861 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903809 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-kubelet\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.903982 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903883 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-slash\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.903982 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903910 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-var-lib-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.903982 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903927 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-bin\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.903982 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903980 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mslsf\" (UniqueName: \"kubernetes.io/projected/6d201227-679c-4342-8bbc-f386d540cc81-kube-api-access-mslsf\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:05.904103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.903998 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.904103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-netns\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904048 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-bin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904063 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a72a542-4dd5-46f1-bdba-e82aba26d948-host\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:05.904103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904079 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e5f527d-4282-4ace-b004-5a2504a1d73e-agent-certs\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904141 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-netd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904160 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-env-overrides\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904178 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904192 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-multus\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904207 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904224 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-conf\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904262 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87sz4\" (UniqueName: \"kubernetes.io/projected/5a72a542-4dd5-46f1-bdba-e82aba26d948-kube-api-access-87sz4\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:05.904288 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904286 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da158b42-504b-4dfa-9b41-3333f4bbf0e7-tmp-dir\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-log-socket\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904331 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904416 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-socket-dir-parent\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904441 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48bzs\" (UniqueName: \"kubernetes.io/projected/aa997701-363d-4c89-98f7-b6e5c349b892-kube-api-access-48bzs\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904459 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-netns\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904474 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-kubelet\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-multus-certs\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfq98\" (UniqueName: \"kubernetes.io/projected/ee24becd-13c7-4fd5-a314-0b5f27a066ad-kube-api-access-mfq98\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904523 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904547 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-kubernetes\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-tuned\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrksh\" (UniqueName: \"kubernetes.io/projected/8d4333e0-6941-4815-844e-b7641b1ae720-kube-api-access-qrksh\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:05.904632 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcfzz\" (UniqueName: \"kubernetes.io/projected/d2e9a5db-d5d7-4b26-828a-203c6557820e-kube-api-access-bcfzz\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904641 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-system-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904680 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-sys\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904697 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da158b42-504b-4dfa-9b41-3333f4bbf0e7-hosts-file\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-script-lib\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904728 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-k8s-cni-cncf-io\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904752 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-hostroot\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904777 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-dbus\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904804 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-node-log\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904928 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovn-node-metrics-cert\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904952 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904971 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-conf-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.904986 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-systemd\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905002 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2k5h\" (UniqueName: \"kubernetes.io/projected/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-kube-api-access-h2k5h\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905019 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905033 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905035 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-ovn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905051 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-config\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xcnn\" (UniqueName: \"kubernetes.io/projected/f178ba1f-4d58-4164-862d-0b99e5ffa859-kube-api-access-4xcnn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905098 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d201227-679c-4342-8bbc-f386d540cc81-host-slash\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905117 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905132 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905148 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905163 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d201227-679c-4342-8bbc-f386d540cc81-iptables-alerter-script\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905181 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cnibin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905211 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a72a542-4dd5-46f1-bdba-e82aba26d948-serviceca\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905233 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-var-lib-kubelet\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-systemd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905287 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysconfig\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905319 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-lib-modules\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905346 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-etc-kubernetes\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.905544 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905366 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905389 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905427 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-run\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905510 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9xx9\" (UniqueName: \"kubernetes.io/projected/da158b42-504b-4dfa-9b41-3333f4bbf0e7-kube-api-access-n9xx9\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905560 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-systemd-units\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905598 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905631 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-device-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905658 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905698 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905881 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-modprobe-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-host\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.905956 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-daemon-config\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.906008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-system-cni-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.906041 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-kubelet-config\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.906070 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f527d-4282-4ace-b004-5a2504a1d73e-konnectivity-ca\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:05.906146 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.906096 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-tmp\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924315 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p94r5\"" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924357 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924383 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924371 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzpht\"" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924384 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924436 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 14:47:05.924452 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924451 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2zmnb\"" Oct 30 14:47:05.924782 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924725 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 14:47:05.924782 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924772 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:47:05.924859 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924803 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 14:47:05.924900 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924858 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.924900 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924876 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 14:47:05.924965 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.924945 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 14:47:05.925136 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.925113 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sf926\"" Oct 30 14:47:05.925261 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.925229 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.925326 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.925272 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.925326 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.925281 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 14:47:05.925326 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.925321 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 14:47:05.925503 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.925326 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 14:47:05.926977 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.926943 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 14:47:05.927294 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.927272 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 14:47:05.927471 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.927444 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 14:47:05.931329 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.931311 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 14:47:05.931455 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.931390 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 14:47:05.931455 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.931438 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b74bf\"" Oct 30 14:47:05.931455 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.931451 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bksxs\"" Oct 30 14:47:05.931623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:05.931485 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 14:47:06.007112 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007047 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a72a542-4dd5-46f1-bdba-e82aba26d948-host\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.007112 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e5f527d-4282-4ace-b004-5a2504a1d73e-agent-certs\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-netd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-env-overrides\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007175 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a72a542-4dd5-46f1-bdba-e82aba26d948-host\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007225 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-multus\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007244 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.007270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007268 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-conf\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007306 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sz4\" (UniqueName: \"kubernetes.io/projected/5a72a542-4dd5-46f1-bdba-e82aba26d948-kube-api-access-87sz4\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007321 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da158b42-504b-4dfa-9b41-3333f4bbf0e7-tmp-dir\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007339 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-log-socket\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007365 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007401 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-socket-dir-parent\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007427 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bzs\" (UniqueName: \"kubernetes.io/projected/aa997701-363d-4c89-98f7-b6e5c349b892-kube-api-access-48bzs\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007451 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-netns\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-kubelet\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007501 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-multus-certs\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfq98\" (UniqueName: \"kubernetes.io/projected/ee24becd-13c7-4fd5-a314-0b5f27a066ad-kube-api-access-mfq98\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.007557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007548 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007574 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-kubernetes\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007600 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-tuned\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007625 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrksh\" (UniqueName: \"kubernetes.io/projected/8d4333e0-6941-4815-844e-b7641b1ae720-kube-api-access-qrksh\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007651 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfzz\" (UniqueName: \"kubernetes.io/projected/d2e9a5db-d5d7-4b26-828a-203c6557820e-kube-api-access-bcfzz\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007679 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-system-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007702 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-sys\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da158b42-504b-4dfa-9b41-3333f4bbf0e7-hosts-file\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007752 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-script-lib\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007777 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-k8s-cni-cncf-io\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007833 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-k8s-cni-cncf-io\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.007880 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-multus\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.008083 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.008076 2571 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 14:47:06.008644 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.008406 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.008644 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.008605 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-conf\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.008890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.008869 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-env-overrides\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.009158 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009101 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da158b42-504b-4dfa-9b41-3333f4bbf0e7-tmp-dir\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.009158 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009137 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-log-socket\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.009301 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009195 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.009301 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009279 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-kubernetes\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.009424 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009299 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-socket-dir-parent\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.009583 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-netns\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.009659 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009615 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-kubelet\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.009715 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009667 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-multus-certs\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.010267 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009836 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-sys\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.010267 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.009917 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-system-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.010267 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010142 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.010267 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010205 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-netd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010267 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010229 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da158b42-504b-4dfa-9b41-3333f4bbf0e7-hosts-file\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010331 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-hostroot\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-dbus\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010417 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-node-log\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010446 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-script-lib\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010441 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovn-node-metrics-cert\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010502 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-conf-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010546 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-systemd\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.010605 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2k5h\" (UniqueName: \"kubernetes.io/projected/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-kube-api-access-h2k5h\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010611 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-ovn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010670 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-config\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xcnn\" (UniqueName: \"kubernetes.io/projected/f178ba1f-4d58-4164-862d-0b99e5ffa859-kube-api-access-4xcnn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010720 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d201227-679c-4342-8bbc-f386d540cc81-host-slash\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010746 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010818 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010857 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010916 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d201227-679c-4342-8bbc-f386d540cc81-iptables-alerter-script\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.010943 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010941 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cnibin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010965 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a72a542-4dd5-46f1-bdba-e82aba26d948-serviceca\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.010991 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-var-lib-kubelet\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011013 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-systemd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011036 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011084 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysconfig\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-lib-modules\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-etc-kubernetes\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011170 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011198 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-run\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011266 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xx9\" (UniqueName: \"kubernetes.io/projected/da158b42-504b-4dfa-9b41-3333f4bbf0e7-kube-api-access-n9xx9\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011291 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-systemd-units\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011353 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-device-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011407 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.011468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011458 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-modprobe-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011525 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-hostroot\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011538 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011544 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-host\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011571 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-conf-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011614 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-systemd\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011642 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-var-lib-kubelet\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011671 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-dbus\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011683 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-systemd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011725 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-node-log\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011778 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.011808 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-ovn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012061 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012120 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysconfig\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012185 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a72a542-4dd5-46f1-bdba-e82aba26d948-serviceca\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012217 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-config\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.012270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012225 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-lib-modules\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012290 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012291 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012347 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-modprobe-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-daemon-config\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.012451 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012497 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012515 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-host\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012544 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-run\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-system-cni-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-kubelet-config\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012593 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-device-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012610 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f527d-4282-4ace-b004-5a2504a1d73e-konnectivity-ca\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012636 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-tmp\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012637 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cnibin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012683 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-etc-kubernetes\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012727 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-etc-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-etc-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013126 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012782 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cni-binary-copy\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012807 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-cnibin\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012838 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-kubelet\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012874 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-slash\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012900 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-var-lib-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-bin\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012994 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslsf\" (UniqueName: \"kubernetes.io/projected/6d201227-679c-4342-8bbc-f386d540cc81-kube-api-access-mslsf\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.012999 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-var-lib-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013054 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-netns\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-bin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013227 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-bin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013261 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-bin\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013358 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d201227-679c-4342-8bbc-f386d540cc81-iptables-alerter-script\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013505 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013549 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-systemd-units\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013710 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f527d-4282-4ace-b004-5a2504a1d73e-konnectivity-ca\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:06.013987 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013789 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-tuned\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013783 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovn-node-metrics-cert\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013875 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013906 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-daemon-config\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013919 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-cnibin\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.013967 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-kubelet-config\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014078 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-system-cni-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.014146 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.514113116 +0000 UTC m=+3.132643035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014166 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d201227-679c-4342-8bbc-f386d540cc81-host-slash\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.014280 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.014330 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.514312987 +0000 UTC m=+3.132842907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014381 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014562 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-slash\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014608 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-kubelet\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014715 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014741 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cni-binary-copy\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.014942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.014778 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-netns\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.016347 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.016319 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e5f527d-4282-4ace-b004-5a2504a1d73e-agent-certs\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:06.017792 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.017773 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-tmp\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.035884 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.035866 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:06.035995 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.035888 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:06.035995 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.035900 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:06.035995 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.035968 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:06.535952722 +0000 UTC m=+3.154482634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:06.037708 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.037678 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sz4\" (UniqueName: \"kubernetes.io/projected/5a72a542-4dd5-46f1-bdba-e82aba26d948-kube-api-access-87sz4\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.038289 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.038264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrksh\" (UniqueName: \"kubernetes.io/projected/8d4333e0-6941-4815-844e-b7641b1ae720-kube-api-access-qrksh\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:06.038289 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.038279 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfzz\" (UniqueName: \"kubernetes.io/projected/d2e9a5db-d5d7-4b26-828a-203c6557820e-kube-api-access-bcfzz\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.040524 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.040502 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfq98\" (UniqueName: \"kubernetes.io/projected/ee24becd-13c7-4fd5-a314-0b5f27a066ad-kube-api-access-mfq98\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.040758 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.040735 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bzs\" (UniqueName: \"kubernetes.io/projected/aa997701-363d-4c89-98f7-b6e5c349b892-kube-api-access-48bzs\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.042383 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.042344 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2k5h\" (UniqueName: \"kubernetes.io/projected/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-kube-api-access-h2k5h\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.042917 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.042893 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xx9\" (UniqueName: \"kubernetes.io/projected/da158b42-504b-4dfa-9b41-3333f4bbf0e7-kube-api-access-n9xx9\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.044819 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.044789 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xcnn\" (UniqueName: \"kubernetes.io/projected/f178ba1f-4d58-4164-862d-0b99e5ffa859-kube-api-access-4xcnn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.046805 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.046783 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslsf\" (UniqueName: \"kubernetes.io/projected/6d201227-679c-4342-8bbc-f386d540cc81-kube-api-access-mslsf\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.112557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.112539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:06.122625 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.122467 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:47:06.125607 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.125535 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e5f527d_4282_4ace_b004_5a2504a1d73e.slice/crio-acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3 WatchSource:0}: Error finding container acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3: Status 404 returned error can't find the container with id acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3 Oct 30 14:47:06.130633 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.130614 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5b49" Oct 30 14:47:06.133089 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.133062 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333a5e10_a1b0_48e3_a2d8_ff1c9221bb8d.slice/crio-cacd84101a534e1c488beadc239e268b17e3c07bc733e378b8b78daa216c6853 WatchSource:0}: Error finding container cacd84101a534e1c488beadc239e268b17e3c07bc733e378b8b78daa216c6853: Status 404 returned error can't find the container with id cacd84101a534e1c488beadc239e268b17e3c07bc733e378b8b78daa216c6853 Oct 30 14:47:06.144363 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.144341 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda158b42_504b_4dfa_9b41_3333f4bbf0e7.slice/crio-af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586 WatchSource:0}: Error finding container af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586: Status 404 returned error can't find the container with id af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586 Oct 30 14:47:06.153854 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.153827 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:47:06.158212 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.158197 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:06.164176 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.164151 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d201227_679c_4342_8bbc_f386d540cc81.slice/crio-3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7 WatchSource:0}: Error finding container 3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7: Status 404 returned error can't find the container with id 3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7 Oct 30 14:47:06.169012 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.168990 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf178ba1f_4d58_4164_862d_0b99e5ffa859.slice/crio-d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693 WatchSource:0}: Error finding container d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693: Status 404 returned error can't find the container with id d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693 Oct 30 14:47:06.172797 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.172772 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:47:06.180229 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.180209 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:47:06.183089 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.183063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hjchw" Oct 30 14:47:06.184001 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.183976 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa997701_363d_4c89_98f7_b6e5c349b892.slice/crio-f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311 WatchSource:0}: Error finding container f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311: Status 404 returned error can't find the container with id f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311 Oct 30 14:47:06.185407 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.185386 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:47:06.193329 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:06.193304 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2e9a5db_d5d7_4b26_828a_203c6557820e.slice/crio-985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25 WatchSource:0}: Error finding container 985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25: Status 404 returned error can't find the container with id 985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25 Oct 30 14:47:06.517771 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.517742 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:06.517893 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.517786 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:06.517949 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.517903 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:06.518004 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.517955 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:06.518004 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.517971 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:07.517955825 +0000 UTC m=+4.136485719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:06.518004 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.517993 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:07.517982637 +0000 UTC m=+4.136512531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:06.619579 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.619243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:06.619579 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.619445 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:06.619579 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.619470 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:06.619579 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.619483 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:06.619579 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:06.619542 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:07.61952199 +0000 UTC m=+4.238051902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:06.880545 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.880438 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:42:06 +0000 UTC" deadline="2028-03-24 21:31:32.485766216 +0000 UTC" Oct 30 14:47:06.880545 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.880502 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12270h44m25.605268313s" Oct 30 14:47:06.944242 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.944191 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hzqxz" event={"ID":"6d201227-679c-4342-8bbc-f386d540cc81","Type":"ContainerStarted","Data":"3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7"} Oct 30 14:47:06.947104 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.947073 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5b49" event={"ID":"da158b42-504b-4dfa-9b41-3333f4bbf0e7","Type":"ContainerStarted","Data":"af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586"} Oct 30 14:47:06.949921 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.949892 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" event={"ID":"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d","Type":"ContainerStarted","Data":"cacd84101a534e1c488beadc239e268b17e3c07bc733e378b8b78daa216c6853"} Oct 30 14:47:06.953696 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.953448 2571 generic.go:356] "Generic (PLEG): container finished" podID="4c8f5d16b96687b2a915ddaffb597231" containerID="ce4e069c6fe2a275f604bb87c661fd7f11a2af0000af8617b4e5c23d2dead5ff" exitCode=0 Oct 30 14:47:06.953814 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.953752 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"4c8f5d16b96687b2a915ddaffb597231","Type":"ContainerDied","Data":"ce4e069c6fe2a275f604bb87c661fd7f11a2af0000af8617b4e5c23d2dead5ff"} Oct 30 14:47:06.970361 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.970334 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjchw" event={"ID":"ee24becd-13c7-4fd5-a314-0b5f27a066ad","Type":"ContainerStarted","Data":"c73bcc3e26c17d123280dceb14916054cdd09f5f3af9f2bd0d2e9e9d0b60c5e6"} Oct 30 14:47:06.975591 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.975563 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerStarted","Data":"f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311"} Oct 30 14:47:06.981747 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.981713 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693"} Oct 30 14:47:06.994743 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:06.994716 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbcnp" event={"ID":"8e5f527d-4282-4ace-b004-5a2504a1d73e","Type":"ContainerStarted","Data":"acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3"} Oct 30 14:47:07.000733 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.000700 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" event={"ID":"c47c2ff730e7510466e8747f4e5f45ca","Type":"ContainerStarted","Data":"9ea763045043bfb5c8b3e741b4b19ad9c84af466ffadba2738928187e0aa9cb5"} Oct 30 14:47:07.012426 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.012394 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvpcd" event={"ID":"5a72a542-4dd5-46f1-bdba-e82aba26d948","Type":"ContainerStarted","Data":"e522e488059a8c2d0ba51f94605e25406cff59318ee036f1e13e3c9414b11de6"} Oct 30 14:47:07.017818 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.017782 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" event={"ID":"d2e9a5db-d5d7-4b26-828a-203c6557820e","Type":"ContainerStarted","Data":"985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25"} Oct 30 14:47:07.534920 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.534802 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:07.534920 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.534854 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:07.535143 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.535011 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:07.535143 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.535069 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:09.535051625 +0000 UTC m=+6.153581541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:07.535143 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.535138 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:07.535333 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.535175 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:09.535161052 +0000 UTC m=+6.153690962 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:07.636131 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.636059 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:07.636328 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.636308 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:07.636396 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.636336 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:07.636396 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.636372 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:07.636536 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.636478 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:09.636457499 +0000 UTC m=+6.254987405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:07.881079 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.880942 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:42:06 +0000 UTC" deadline="2028-04-14 02:55:11.023821507 +0000 UTC" Oct 30 14:47:07.881079 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.881000 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12756h8m3.142826354s" Oct 30 14:47:07.940463 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.940278 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:07.940463 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.940383 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:07.940670 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.940494 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:07.940670 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.940569 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:07.940670 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:07.940647 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:07.940837 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:07.940720 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:08.035133 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:08.035036 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"4c8f5d16b96687b2a915ddaffb597231","Type":"ContainerStarted","Data":"46c8575b097b7a12545892f8e3fc4e1084cadb08d422b2af0c400a66c9af8f5c"} Oct 30 14:47:08.080089 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:08.079483 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" podStartSLOduration=2.079468392 podStartE2EDuration="2.079468392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:07.066830062 +0000 UTC m=+3.685359983" watchObservedRunningTime="2026-10-30 14:47:08.079468392 +0000 UTC m=+4.697998311" Oct 30 14:47:09.044390 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.044204 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="8460970d5660af7f41a8f6f4c79d342700c925263ed8b362f102160623f84d3a" exitCode=0 Oct 30 14:47:09.044885 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.044501 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerDied","Data":"8460970d5660af7f41a8f6f4c79d342700c925263ed8b362f102160623f84d3a"} Oct 30 14:47:09.124477 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.124396 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" podStartSLOduration=3.124376595 podStartE2EDuration="3.124376595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:08.080067458 +0000 UTC m=+4.698597375" watchObservedRunningTime="2026-10-30 14:47:09.124376595 +0000 UTC m=+5.742906575" Oct 30 14:47:09.556616 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.555811 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:09.556616 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.555862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:09.556616 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.555995 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:09.556616 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.556064 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:13.556043171 +0000 UTC m=+10.174573069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:09.556616 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.556517 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:09.556616 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.556572 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:13.556554593 +0000 UTC m=+10.175084506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:09.657790 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.657145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:09.657790 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.657319 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:09.657790 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.657342 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:09.657790 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.657357 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:09.657790 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.657430 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:13.657410048 +0000 UTC m=+10.275939961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:09.935628 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.935544 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:09.935810 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.935702 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:09.936051 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.936026 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:09.936198 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.936149 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:09.936273 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:09.936208 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:09.936330 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:09.936298 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:11.054265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:11.053444 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sbcnp" event={"ID":"8e5f527d-4282-4ace-b004-5a2504a1d73e","Type":"ContainerStarted","Data":"c6d466250f71de3e82eb19e3f3ca03050d733c5998f45f24ed3588766f20b0d4"} Oct 30 14:47:11.934578 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:11.934499 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:11.934731 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:11.934650 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:11.935088 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:11.935062 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:11.935193 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:11.935167 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:11.935443 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:11.935423 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:11.935550 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:11.935528 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:12.058658 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:12.058620 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vvpcd" event={"ID":"5a72a542-4dd5-46f1-bdba-e82aba26d948","Type":"ContainerStarted","Data":"a76a4330e78329c2c6e64cefd17ebed154ae9b17ea702196b29fa0dca190e3a8"} Oct 30 14:47:12.102390 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:12.100986 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sbcnp" podStartSLOduration=2.919607914 podStartE2EDuration="7.100966711s" totalImagesPullingTime="4.181358797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:11.101314345 +0000 UTC m=+7.719844263" watchObservedRunningTime="2026-10-30 14:47:12.100966711 +0000 UTC m=+8.719496626" Oct 30 14:47:12.102390 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:12.101589 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vvpcd" podStartSLOduration=1.812961902 podStartE2EDuration="7.101570822s" totalImagesPullingTime="5.28860892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:12.100942852 +0000 UTC m=+8.719472769" watchObservedRunningTime="2026-10-30 14:47:12.101570822 +0000 UTC m=+8.720100743" Oct 30 14:47:13.066224 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.066138 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" event={"ID":"d2e9a5db-d5d7-4b26-828a-203c6557820e","Type":"ContainerStarted","Data":"215b7bcd737698b0e8535f103e442cde78012a4e216f37bc437a60f180bb6f95"} Oct 30 14:47:13.593766 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.592963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:13.593766 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.593015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:13.593766 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.593184 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:13.593766 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.593246 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:21.593228799 +0000 UTC m=+18.211758711 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:13.593766 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.593322 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:13.593766 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.593358 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:21.593345829 +0000 UTC m=+18.211875752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:13.694364 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.694311 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:13.694618 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.694597 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:13.694699 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.694627 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:13.694699 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.694640 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:13.694809 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.694703 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:21.694684824 +0000 UTC m=+18.313214722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:13.934438 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.934190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:13.934438 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.934328 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:13.936436 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.934738 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:13.936436 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.934850 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:13.936436 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:13.934913 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:13.936436 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:13.935004 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:14.061482 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:14.061444 2571 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 14:47:14.070724 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:14.070691 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" event={"ID":"d2e9a5db-d5d7-4b26-828a-203c6557820e","Type":"ContainerStarted","Data":"06d1e0781c1b86161b2836a06ecbba2199c26d3cc4a965be76d3ba2ea85eb846"} Oct 30 14:47:15.000123 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:14.999891 2571 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:47:14.061466246Z","UUID":"bb5ae7ad-ac7e-4c0b-b6ad-e368a52f0b3c","Handler":null,"Name":"","Endpoint":""} Oct 30 14:47:15.002261 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.001866 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 14:47:15.002261 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.001914 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 14:47:15.075566 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.075526 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="ee2bebe6bcd0ced81a8d5a5c8cbc4aefee9ca1d6385d8ad6b3dceb714ecfbd1d" exitCode=0 Oct 30 14:47:15.076061 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.075637 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerDied","Data":"ee2bebe6bcd0ced81a8d5a5c8cbc4aefee9ca1d6385d8ad6b3dceb714ecfbd1d"} Oct 30 14:47:15.082295 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.082264 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5b49" event={"ID":"da158b42-504b-4dfa-9b41-3333f4bbf0e7","Type":"ContainerStarted","Data":"2fdbbf8bffba1a62dd25dd7d7e723ff5f5d1239ff000f7104cb3e8d0e235fb0e"} Oct 30 14:47:15.084541 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.084497 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hjchw" event={"ID":"ee24becd-13c7-4fd5-a314-0b5f27a066ad","Type":"ContainerStarted","Data":"41c33bdf95ed22a2271a9a35a1a8395c57be76f0d335d77438deda66b5e157a9"} Oct 30 14:47:15.171767 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.171515 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q5b49" podStartSLOduration=1.6705098710000001 podStartE2EDuration="10.171496597s" totalImagesPullingTime="8.500986726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:15.170579633 +0000 UTC m=+11.789109550" watchObservedRunningTime="2026-10-30 14:47:15.171496597 +0000 UTC m=+11.790026515" Oct 30 14:47:15.218557 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.218499 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-hjchw" podStartSLOduration=2.062690113 podStartE2EDuration="10.218481074s" totalImagesPullingTime="8.155790961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:15.217425262 +0000 UTC m=+11.835955184" watchObservedRunningTime="2026-10-30 14:47:15.218481074 +0000 UTC m=+11.837010996" Oct 30 14:47:15.645249 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.645162 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:15.646261 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.646234 2571 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:15.934283 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.933757 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:15.934283 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.933790 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:15.934283 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:15.933765 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:15.934283 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:15.933897 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:15.934283 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:15.933979 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:15.934283 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:15.934056 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:16.095117 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.095086 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="87e58bb1cf14d4d3c21e2d9f06c85438cc899b9c84fae5ae726774d96f1fb4d4" exitCode=0 Oct 30 14:47:16.095617 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.095153 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerDied","Data":"87e58bb1cf14d4d3c21e2d9f06c85438cc899b9c84fae5ae726774d96f1fb4d4"} Oct 30 14:47:16.098937 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.098908 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" event={"ID":"d2e9a5db-d5d7-4b26-828a-203c6557820e","Type":"ContainerStarted","Data":"9df674799b748a830bd864597fb0820f6865b1797377730c6c98a9dce8b1f016"} Oct 30 14:47:16.101839 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.101742 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hzqxz" event={"ID":"6d201227-679c-4342-8bbc-f386d540cc81","Type":"ContainerStarted","Data":"46de0462fc78315a0b8a95e7d55796daebe2388b75f321a272dfa17714f0422d"} Oct 30 14:47:16.101839 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.101795 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:16.102552 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.102387 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:47:16.189621 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.189514 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hzqxz" podStartSLOduration=2.885015074 podStartE2EDuration="11.189495215s" totalImagesPullingTime="8.304480141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:16.188745016 +0000 UTC m=+12.807274935" watchObservedRunningTime="2026-10-30 14:47:16.189495215 +0000 UTC m=+12.808025134" Oct 30 14:47:16.235564 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:16.235501 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" podStartSLOduration=2.667359827 podStartE2EDuration="11.235483316s" totalImagesPullingTime="8.568123489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:16.235288324 +0000 UTC m=+12.853818243" watchObservedRunningTime="2026-10-30 14:47:16.235483316 +0000 UTC m=+12.854013235" Oct 30 14:47:17.105895 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.105861 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" event={"ID":"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d","Type":"ContainerStarted","Data":"9f96c9b7325f81e486cfbb67e70c6cd8cc5b3e85bb870c248be374a239f17eef"} Oct 30 14:47:17.108585 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.108559 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="348eec614aecf18363d407d494b066af89d921e33d60c889107e7381c2e85487" exitCode=0 Oct 30 14:47:17.108711 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.108621 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerDied","Data":"348eec614aecf18363d407d494b066af89d921e33d60c889107e7381c2e85487"} Oct 30 14:47:17.152305 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.152265 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" podStartSLOduration=1.618661101 podStartE2EDuration="12.152253637s" totalImagesPullingTime="10.533592536s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:17.151785898 +0000 UTC m=+13.770315813" watchObservedRunningTime="2026-10-30 14:47:17.152253637 +0000 UTC m=+13.770783550" Oct 30 14:47:17.935577 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.935536 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:17.935751 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:17.935660 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:17.936057 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.936039 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:17.936159 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:17.936140 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:17.936224 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:17.936209 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:17.936311 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:17.936292 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:19.115942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:19.115716 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="faa028b33643f77b6a9776aedf98dda1605cb09ec3de2a098cc8c62e02693f67" exitCode=0 Oct 30 14:47:19.115942 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:19.115806 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerDied","Data":"faa028b33643f77b6a9776aedf98dda1605cb09ec3de2a098cc8c62e02693f67"} Oct 30 14:47:19.933565 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:19.933523 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:19.933565 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:19.933535 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:19.933781 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:19.933701 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:19.934012 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:19.933991 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:19.934148 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:19.934088 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:19.934213 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:19.934163 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:20.121461 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:20.121427 2571 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="760bbb81d9d24c37246230c8d22687c16f60f118ecdfa5039f46c087ef602fce" exitCode=0 Oct 30 14:47:20.121867 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:20.121460 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerDied","Data":"760bbb81d9d24c37246230c8d22687c16f60f118ecdfa5039f46c087ef602fce"} Oct 30 14:47:21.126249 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.126216 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" event={"ID":"aa997701-363d-4c89-98f7-b6e5c349b892","Type":"ContainerStarted","Data":"0152081aaedaf73d9db34f34ce94e6f3244a98453c2e360083d37d01e5bb2ae5"} Oct 30 14:47:21.202329 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.202283 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rgsm9" podStartSLOduration=5.869369279 podStartE2EDuration="16.202271053s" totalImagesPullingTime="10.332901774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:21.201838146 +0000 UTC m=+17.820368060" watchObservedRunningTime="2026-10-30 14:47:21.202271053 +0000 UTC m=+17.820800963" Oct 30 14:47:21.657262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.657098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:21.657262 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.657235 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:21.657262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.657247 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:21.657439 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.657297 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:37.65727693 +0000 UTC m=+34.275806825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:21.657439 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.657323 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:21.657439 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.657383 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:37.657371377 +0000 UTC m=+34.275901285 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:21.758313 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.758290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:21.758445 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.758433 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:21.758483 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.758449 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:21.758483 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.758457 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:21.758548 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.758495 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:47:37.758484184 +0000 UTC m=+34.377014089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:21.934322 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.934259 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:21.934322 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.934289 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:21.934509 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:21.934392 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:21.934558 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.934521 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:21.934661 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.934639 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:21.934727 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:21.934712 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:22.131440 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:22.131413 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"10f0523d0ee11d2409b7a8f59d1771effb0056fd2e2a798d6c83be4d7d8dec1f"} Oct 30 14:47:22.131755 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:22.131477 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"6ee65dd9b88c25b9ab3513a690144286201a2543ae5a08696e87cb2e86259886"} Oct 30 14:47:22.131755 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:22.131510 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"25e44c718d54475176a8847be73662dd100cc320f8a4cb82c1d0037f1857ce04"} Oct 30 14:47:22.131755 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:22.131537 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"ab464fa824487140b284f3a0c75db9064392886e90dd8dad100f718cf96fccb3"} Oct 30 14:47:22.131755 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:22.131564 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"5402bc3b1e63fae03b696d98fe8c0b0ed0c1dca4aba0662e92ea5ee9db5e92ce"} Oct 30 14:47:22.131755 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:22.131592 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"75fc6f7d1ff5ded6c5f6a582d616c588780c6d5148ea81eacb7163d17d65a4b4"} Oct 30 14:47:23.937215 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:23.935936 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:23.937215 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:23.936088 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:23.937215 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:23.936484 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:23.937215 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:23.936601 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:23.937215 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:23.936653 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:23.937215 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:23.936754 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:24.138554 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:24.138525 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"1c543672016786ee0122c2c931399fe13506f72e9c2f2c094f61ce623e9f13eb"} Oct 30 14:47:25.933453 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:25.933422 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:25.934130 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:25.933532 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:25.934130 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:25.933562 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:25.934130 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:25.933682 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:25.934130 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:25.933715 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:25.934130 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:25.933805 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:26.145410 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:26.145329 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" event={"ID":"f178ba1f-4d58-4164-862d-0b99e5ffa859","Type":"ContainerStarted","Data":"9ab09cfb02d6a30f15f7eef179b87edc97b8a6a29a49a9089cfcfbece9781f87"} Oct 30 14:47:26.145737 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:26.145711 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:26.145835 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:26.145792 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:26.160925 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:26.160904 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:26.247553 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:26.247506 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" podStartSLOduration=6.087449397 podStartE2EDuration="21.247494232s" totalImagesPullingTime="15.160044835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:26.233807966 +0000 UTC m=+22.852337880" watchObservedRunningTime="2026-10-30 14:47:26.247494232 +0000 UTC m=+22.866024146" Oct 30 14:47:27.147694 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.147659 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:27.162486 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.162462 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:27.923166 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.922967 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fnbqw"] Oct 30 14:47:27.923321 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.923213 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:27.923321 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:27.923300 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:27.925797 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.925747 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cjwh8"] Oct 30 14:47:27.925901 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.925881 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:27.926026 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:27.926003 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:27.926289 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.926269 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bz4fj"] Oct 30 14:47:27.926380 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:27.926372 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:27.926464 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:27.926447 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:29.253962 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:29.253909 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5b49_da158b42-504b-4dfa-9b41-3333f4bbf0e7/dns-node-resolver/0.log" Oct 30 14:47:29.933889 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:29.933863 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:29.933889 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:29.933885 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:29.934054 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:29.933923 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:29.934095 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:29.934039 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:29.934171 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:29.934152 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:29.934237 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:29.934222 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:30.392493 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:30.392472 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvpcd_5a72a542-4dd5-46f1-bdba-e82aba26d948/node-ca/0.log" Oct 30 14:47:31.933720 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:31.933689 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:31.934162 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:31.933726 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:31.934162 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:31.933702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:31.934162 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:31.933815 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:31.934162 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:31.933870 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:31.934162 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:31.933953 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:33.934968 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:33.934935 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:33.935295 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:33.935053 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:33.935295 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:33.935105 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:33.935295 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:33.935159 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:33.935295 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:33.935239 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:33.935439 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:33.935308 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:35.934140 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:35.934103 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:35.934140 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:35.934119 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:35.934664 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:35.934238 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:35.934664 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:35.934373 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:35.934664 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:35.934406 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:35.934664 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:35.934510 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:37.675709 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:37.675676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:37.675709 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:37.675710 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:37.676131 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.675806 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:37.676131 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.675836 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:37.676131 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.675858 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret podName:1c36de38-25e2-4eb1-9b01-a7c342084792 nodeName:}" failed. No retries permitted until 2026-10-30 14:48:09.67584492 +0000 UTC m=+66.294374812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret") pod "global-pull-secret-syncer-bz4fj" (UID: "1c36de38-25e2-4eb1-9b01-a7c342084792") : object "kube-system"/"original-pull-secret" not registered Oct 30 14:47:37.676131 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.675901 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs podName:8d4333e0-6941-4815-844e-b7641b1ae720 nodeName:}" failed. No retries permitted until 2026-10-30 14:48:09.675884265 +0000 UTC m=+66.294414213 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs") pod "network-metrics-daemon-cjwh8" (UID: "8d4333e0-6941-4815-844e-b7641b1ae720") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 14:47:37.776132 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:37.776104 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:37.776274 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.776247 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 14:47:37.776274 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.776268 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 14:47:37.776389 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.776277 2571 projected.go:196] Error preparing data for projected volume kube-api-access-7nzhv for pod openshift-network-diagnostics/network-check-target-fnbqw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:37.776389 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.776319 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv podName:4f8be0f7-d4c4-4bc9-ae48-ef845d362377 nodeName:}" failed. No retries permitted until 2026-10-30 14:48:09.776308465 +0000 UTC m=+66.394838358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7nzhv" (UniqueName: "kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv") pod "network-check-target-fnbqw" (UID: "4f8be0f7-d4c4-4bc9-ae48-ef845d362377") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 14:47:37.933655 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:37.933601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:37.933655 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:37.933624 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:37.933782 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.933697 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:37.933782 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:37.933743 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:37.933865 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.933819 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:37.933945 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:37.933923 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:39.933968 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:39.933930 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:39.934527 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:39.933930 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:39.934527 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:39.934071 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:39.934527 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:39.933933 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:39.934527 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:39.934141 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:39.934527 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:39.934225 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:41.933963 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:41.933925 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:41.934484 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:41.933967 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:41.934484 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:41.934024 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:41.934484 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:41.934135 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:41.934484 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:41.934224 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:41.934484 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:41.934386 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:43.936446 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:43.936416 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:43.936799 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:43.936506 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:43.936799 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:43.936516 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:43.936799 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:43.936533 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:43.936799 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:43.936597 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:43.936799 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:43.936671 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:45.936154 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:45.936126 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:45.936154 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:45.936151 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:45.936560 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:45.936223 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:45.936560 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:45.936286 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:45.936560 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:45.936385 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:45.936560 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:45.936426 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:47.933802 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:47.933766 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:47.934292 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:47.933775 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:47.934292 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:47.933880 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:47.934292 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:47.934014 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:47.934292 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:47.934025 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:47.934292 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:47.934168 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:49.933580 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:49.933546 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:49.933580 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:49.933568 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:49.933938 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:49.933655 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fnbqw" podUID="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" Oct 30 14:47:49.933938 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:49.933794 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cjwh8" podUID="8d4333e0-6941-4815-844e-b7641b1ae720" Oct 30 14:47:49.933938 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:49.933834 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:49.933938 ip-10-0-139-204 kubenswrapper[2571]: E1030 14:47:49.933907 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bz4fj" podUID="1c36de38-25e2-4eb1-9b01-a7c342084792" Oct 30 14:47:50.177230 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.177209 2571 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeReady" Oct 30 14:47:50.177369 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.177297 2571 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 14:47:50.245237 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.245215 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xmw9l"] Oct 30 14:47:50.248018 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.248003 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.252394 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.252374 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hj6kg"] Oct 30 14:47:50.255202 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.255184 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.267509 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267490 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-metrics-tls\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.267597 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267532 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-crio-socket\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.267597 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267559 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.267731 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267634 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-tmp-dir\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.267731 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-data-volume\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.267731 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hp9b\" (UniqueName: \"kubernetes.io/projected/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-api-access-5hp9b\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.267731 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267696 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp8wt\" (UniqueName: \"kubernetes.io/projected/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-kube-api-access-cp8wt\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.267891 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-config-volume\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.267891 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.267840 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.273857 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.273835 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmw9l"] Oct 30 14:47:50.274309 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.274281 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mnl72"] Oct 30 14:47:50.277263 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.277245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.278461 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.278442 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hj6kg"] Oct 30 14:47:50.284303 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.284288 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 14:47:50.284506 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.284489 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 14:47:50.284584 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.284506 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 14:47:50.284629 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.284599 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 14:47:50.285159 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.285145 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cd47w\"" Oct 30 14:47:50.285409 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.285394 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ss7nc\"" Oct 30 14:47:50.285938 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.285924 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 14:47:50.286005 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.285993 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 14:47:50.292861 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.292845 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 14:47:50.293098 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.293087 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b4vks\"" Oct 30 14:47:50.293140 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.293090 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 14:47:50.296185 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.296171 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 14:47:50.297051 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.297034 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnl72"] Oct 30 14:47:50.369010 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.368985 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-metrics-tls\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.369107 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-crio-socket\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369107 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369069 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369107 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369103 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b36b9d5-51b0-4038-ae52-a7d1299cd284-cert\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.369265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369128 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-tmp-dir\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.369265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369153 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-data-volume\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369174 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-crio-socket\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369236 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp9b\" (UniqueName: \"kubernetes.io/projected/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-api-access-5hp9b\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369531 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369271 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8wt\" (UniqueName: \"kubernetes.io/projected/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-kube-api-access-cp8wt\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.369531 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369332 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-config-volume\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.369531 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369531 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369435 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgmz\" (UniqueName: \"kubernetes.io/projected/2b36b9d5-51b0-4038-ae52-a7d1299cd284-kube-api-access-7mgmz\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.369717 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369588 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-tmp-dir\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.369772 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369738 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-data-volume\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369924 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369903 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.369924 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.369916 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-config-volume\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.372454 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.372435 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.372534 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.372438 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-metrics-tls\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.392859 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.392835 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8wt\" (UniqueName: \"kubernetes.io/projected/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-kube-api-access-cp8wt\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.393062 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.393043 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp9b\" (UniqueName: \"kubernetes.io/projected/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-api-access-5hp9b\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.469660 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.469615 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b36b9d5-51b0-4038-ae52-a7d1299cd284-cert\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.469660 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.469658 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgmz\" (UniqueName: \"kubernetes.io/projected/2b36b9d5-51b0-4038-ae52-a7d1299cd284-kube-api-access-7mgmz\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.471598 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.471577 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b36b9d5-51b0-4038-ae52-a7d1299cd284-cert\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.493284 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.493265 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgmz\" (UniqueName: \"kubernetes.io/projected/2b36b9d5-51b0-4038-ae52-a7d1299cd284-kube-api-access-7mgmz\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.562238 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.562215 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:50.570248 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.570231 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:47:50.588106 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.588082 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:47:50.880179 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.880111 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmw9l"] Oct 30 14:47:50.882955 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:50.882924 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d07af5_d2a4_401e_9d97_9b98c4f6e2d8.slice/crio-8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333 WatchSource:0}: Error finding container 8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333: Status 404 returned error can't find the container with id 8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333 Oct 30 14:47:50.896804 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.896786 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hj6kg"] Oct 30 14:47:50.898905 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:50.898883 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e74bcbe_9984_4bc1_bff3_a19c3b0adb50.slice/crio-df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f WatchSource:0}: Error finding container df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f: Status 404 returned error can't find the container with id df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f Oct 30 14:47:50.910711 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:50.910692 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mnl72"] Oct 30 14:47:50.922244 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:50.922224 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b36b9d5_51b0_4038_ae52_a7d1299cd284.slice/crio-24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310 WatchSource:0}: Error finding container 24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310: Status 404 returned error can't find the container with id 24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310 Oct 30 14:47:51.198512 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.198484 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnl72" event={"ID":"2b36b9d5-51b0-4038-ae52-a7d1299cd284","Type":"ContainerStarted","Data":"24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310"} Oct 30 14:47:51.199655 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.199630 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmw9l" event={"ID":"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8","Type":"ContainerStarted","Data":"8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333"} Oct 30 14:47:51.200933 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.200913 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hj6kg" event={"ID":"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50","Type":"ContainerStarted","Data":"a4d1d96f95d8b6a7af8643c11436529f4bfc114a540b7665f841ec5b8b5b45ad"} Oct 30 14:47:51.201002 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.200954 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hj6kg" event={"ID":"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50","Type":"ContainerStarted","Data":"df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f"} Oct 30 14:47:51.934238 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.934207 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:47:51.934410 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.934272 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:47:51.934410 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.934280 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:47:51.950649 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.950629 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:47:51.963512 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.963492 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:47:51.963634 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.963620 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:47:51.963807 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.963787 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\"" Oct 30 14:47:51.976688 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.976663 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:47:51.976799 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.976698 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\"" Oct 30 14:47:51.976799 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:51.976710 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\"" Oct 30 14:47:52.206329 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:52.206243 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hj6kg" event={"ID":"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50","Type":"ContainerStarted","Data":"b3907862420dc70bf5c278d7bd3d12cbbf56c317d8f4b18935ae7c9e79323d47"} Oct 30 14:47:52.697314 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:52.697222 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5b49_da158b42-504b-4dfa-9b41-3333f4bbf0e7/dns-node-resolver/0.log" Oct 30 14:47:53.210989 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:53.210960 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmw9l" event={"ID":"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8","Type":"ContainerStarted","Data":"a4ad6a51454f4b0f95a4f1ead03a0ee76e0c7c2b42c043bce3175f95bc663cef"} Oct 30 14:47:53.211393 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:53.211025 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmw9l" event={"ID":"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8","Type":"ContainerStarted","Data":"6515c38c69717a7b06d03641279eb6d4ecc05565e668987e403e334f95864d42"} Oct 30 14:47:53.211393 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:53.211171 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xmw9l" Oct 30 14:47:53.262494 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:53.262446 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xmw9l" podStartSLOduration=1.5439506 podStartE2EDuration="3.262428396s" totalImagesPullingTime="1.718477796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:53.261883264 +0000 UTC m=+49.880413176" watchObservedRunningTime="2026-10-30 14:47:53.262428396 +0000 UTC m=+49.880958518" Oct 30 14:47:54.102518 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:54.102496 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvpcd_5a72a542-4dd5-46f1-bdba-e82aba26d948/node-ca/0.log" Oct 30 14:47:54.214800 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:54.214769 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hj6kg" event={"ID":"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50","Type":"ContainerStarted","Data":"b86343288f61823b085acfe33bf56511ff8827353422ba32227231ac5ad75bfe"} Oct 30 14:47:54.216420 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:54.216394 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mnl72" event={"ID":"2b36b9d5-51b0-4038-ae52-a7d1299cd284","Type":"ContainerStarted","Data":"444e36f2590b01bf13d012aeb42dec1f515e8d636c2a9584b38166a2fa6ba904"} Oct 30 14:47:54.276702 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:54.276662 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hj6kg" podStartSLOduration=2.168063681 podStartE2EDuration="4.276650425s" totalImagesPullingTime="2.108586744s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:54.276081563 +0000 UTC m=+50.894611477" watchObservedRunningTime="2026-10-30 14:47:54.276650425 +0000 UTC m=+50.895180339" Oct 30 14:47:54.320122 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:54.320082 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mnl72" podStartSLOduration=1.739140688 podStartE2EDuration="4.32006881s" totalImagesPullingTime="2.580928122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:47:54.320042499 +0000 UTC m=+50.938572426" watchObservedRunningTime="2026-10-30 14:47:54.32006881 +0000 UTC m=+50.938598724" Oct 30 14:47:54.698888 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:54.698859 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mnl72_2b36b9d5-51b0-4038-ae52-a7d1299cd284/serve-healthcheck-canary/0.log" Oct 30 14:47:57.938150 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.938106 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8829w"] Oct 30 14:47:57.941492 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.941467 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:57.959818 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.959787 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 14:47:57.959818 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.959799 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 14:47:57.960253 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.960230 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 14:47:57.972625 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.972600 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 14:47:57.972812 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.972795 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-26d46\"" Oct 30 14:47:57.972914 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.972888 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 14:47:57.973990 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:57.973970 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8829w"] Oct 30 14:47:58.005538 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.005506 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw"] Oct 30 14:47:58.008541 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.008526 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.022742 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.022713 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbcn\" (UniqueName: \"kubernetes.io/projected/0a396a10-b996-4d83-9783-f4a6a543ff6e-kube-api-access-rhbcn\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.022832 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.022760 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.022832 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.022812 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a396a10-b996-4d83-9783-f4a6a543ff6e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.022912 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.022862 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.028229 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.028194 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jc64h\"" Oct 30 14:47:58.029858 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.029843 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 14:47:58.030017 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.030005 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 14:47:58.034519 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.034497 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw"] Oct 30 14:47:58.044429 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.044404 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jkbnv"] Oct 30 14:47:58.047284 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.047269 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.059166 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.059149 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 14:47:58.062609 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.062587 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 14:47:58.062747 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.062735 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 14:47:58.062867 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.062857 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7mvkp\"" Oct 30 14:47:58.069785 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.069770 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 14:47:58.123176 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123152 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.123262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123178 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.123262 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123200 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbcn\" (UniqueName: \"kubernetes.io/projected/0a396a10-b996-4d83-9783-f4a6a543ff6e-kube-api-access-rhbcn\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.123394 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123264 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-textfile\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123394 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123288 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123394 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123315 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a396a10-b996-4d83-9783-f4a6a543ff6e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.123394 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123340 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-tls\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123394 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123378 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-sys\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123399 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9lcg\" (UniqueName: \"kubernetes.io/projected/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-kube-api-access-b9lcg\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123415 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123470 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123488 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123505 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prsvj\" (UniqueName: \"kubernetes.io/projected/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-api-access-prsvj\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.123623 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-root\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123846 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123628 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-wtmp\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123846 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.123846 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-metrics-client-ca\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.123846 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.123960 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.123892 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a396a10-b996-4d83-9783-f4a6a543ff6e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.125968 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.125948 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.126042 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.125979 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.152287 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.152265 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-748747647-9rm5d"] Oct 30 14:47:58.155103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.155089 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.157851 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.157832 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbcn\" (UniqueName: \"kubernetes.io/projected/0a396a10-b996-4d83-9783-f4a6a543ff6e-kube-api-access-rhbcn\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.176115 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.176098 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 14:47:58.176202 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.176136 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 14:47:58.176202 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.176147 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ng4tk\"" Oct 30 14:47:58.177990 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.177977 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 14:47:58.178086 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.178070 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 14:47:58.178177 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.178167 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 14:47:58.178222 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.178186 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 14:47:58.178222 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.178205 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 14:47:58.191526 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.191479 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 14:47:58.192219 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.192203 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748747647-9rm5d"] Oct 30 14:47:58.224096 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224071 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lcg\" (UniqueName: \"kubernetes.io/projected/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-kube-api-access-b9lcg\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224173 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224107 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224173 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224137 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224173 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224292 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224204 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224292 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224292 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224255 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224292 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224281 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsvj\" (UniqueName: \"kubernetes.io/projected/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-api-access-prsvj\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224503 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224503 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224400 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-root\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224503 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-wtmp\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224661 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224661 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224534 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-root\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224661 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-metrics-client-ca\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224661 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224578 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224661 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224633 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224742 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.224890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-wtmp\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.224890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224815 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.224890 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224872 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224933 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224947 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-textfile\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.224978 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225015 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-tls\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225042 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-sys\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225047 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-metrics-client-ca\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.225171 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225161 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-sys\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225568 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225250 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-textfile\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.225568 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.225463 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.227173 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.227149 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.227294 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.227273 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.227503 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.227485 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-tls\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.227573 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.227522 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.247158 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.247139 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lcg\" (UniqueName: \"kubernetes.io/projected/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-kube-api-access-b9lcg\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.247477 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.247461 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsvj\" (UniqueName: \"kubernetes.io/projected/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-api-access-prsvj\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.257309 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.257296 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:47:58.324921 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.323815 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:47:58.325478 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325451 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325594 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325533 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325594 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325561 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325594 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325583 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325748 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325647 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325748 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325676 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325748 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325701 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.325748 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.325732 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.329451 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.329424 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.330803 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.330756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.330894 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.330823 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.331438 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.331395 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.331776 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.331739 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.338101 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.338070 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.338786 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.338765 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.353185 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.353160 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.360939 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.360920 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:47:58.372206 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:58.372185 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e2a6a8_11f0_49bd_a0ac_1bcf094bd076.slice/crio-549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088 WatchSource:0}: Error finding container 549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088: Status 404 returned error can't find the container with id 549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088 Oct 30 14:47:58.469778 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.469713 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748747647-9rm5d" Oct 30 14:47:58.581129 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.581104 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8829w"] Oct 30 14:47:58.583612 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:58.583588 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a396a10_b996_4d83_9783_f4a6a543ff6e.slice/crio-fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0 WatchSource:0}: Error finding container fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0: Status 404 returned error can't find the container with id fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0 Oct 30 14:47:58.627604 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.627583 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw"] Oct 30 14:47:58.629504 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:58.629471 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcea1b1f_2eb6_466b_b72f_b1e7d6e50117.slice/crio-df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d WatchSource:0}: Error finding container df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d: Status 404 returned error can't find the container with id df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d Oct 30 14:47:58.766204 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:58.766141 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748747647-9rm5d"] Oct 30 14:47:58.769489 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:47:58.769462 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a21bd0_b038_400e_8e85_be20c12f8ec0.slice/crio-cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d WatchSource:0}: Error finding container cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d: Status 404 returned error can't find the container with id cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d Oct 30 14:47:59.164280 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.164257 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:47:59.231858 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.231766 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" event={"ID":"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117","Type":"ContainerStarted","Data":"df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d"} Oct 30 14:47:59.234496 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.234344 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbnv" event={"ID":"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076","Type":"ContainerStarted","Data":"549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088"} Oct 30 14:47:59.237317 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.237288 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" event={"ID":"0a396a10-b996-4d83-9783-f4a6a543ff6e","Type":"ContainerStarted","Data":"33ba952dabaad6b00a2397373c6ca6a4ea79c99fd6d6ee4e2eb9c380e4d29801"} Oct 30 14:47:59.237424 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.237365 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" event={"ID":"0a396a10-b996-4d83-9783-f4a6a543ff6e","Type":"ContainerStarted","Data":"c3e721d3b4262357709ba38d6d5300478157e6365db7aec43c544b9f85cce236"} Oct 30 14:47:59.237424 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.237399 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" event={"ID":"0a396a10-b996-4d83-9783-f4a6a543ff6e","Type":"ContainerStarted","Data":"fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0"} Oct 30 14:47:59.239019 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.238997 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748747647-9rm5d" event={"ID":"88a21bd0-b038-400e-8e85-be20c12f8ec0","Type":"ContainerStarted","Data":"cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d"} Oct 30 14:47:59.464638 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.463413 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:47:59.467296 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.466779 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.485405 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.484713 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 14:47:59.485405 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.484815 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 14:47:59.485405 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.485266 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 14:47:59.485601 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.485546 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 14:47:59.485669 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.485634 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 14:47:59.485887 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.485866 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 14:47:59.485971 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.485914 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 14:47:59.485971 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.485950 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 14:47:59.486073 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.486033 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ndmfj\"" Oct 30 14:47:59.497737 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.497693 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 14:47:59.499081 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.499063 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:47:59.534580 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534685 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534588 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534685 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534685 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534639 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534685 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534654 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534685 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534670 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534685 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534687 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534727 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534772 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534793 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.534945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.534830 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635499 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635641 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635508 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635641 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635539 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635641 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635587 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635641 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635613 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635641 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635637 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635829 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635659 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635829 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635688 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635829 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635829 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635753 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635829 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635979 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635829 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.635979 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.635861 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.636118 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.636098 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.637114 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.637087 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.637480 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.637420 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.639282 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.639250 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.639852 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.639827 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.643535 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.643510 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.643617 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.643530 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.644325 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.644301 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.644640 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.644619 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.646024 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.645976 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.646847 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.646659 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.646847 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.646810 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.662148 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.662108 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:47:59.794419 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:47:59.794228 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:00.144821 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.144795 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:00.164961 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:00.161165 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393a4718_b786_4ac2_b4b1_cdb250c57261.slice/crio-162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d WatchSource:0}: Error finding container 162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d: Status 404 returned error can't find the container with id 162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d Oct 30 14:48:00.246232 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.246201 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" event={"ID":"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117","Type":"ContainerStarted","Data":"1c37e1c4cee2d6bad106bca00bf97e784f97895bb3c135ce4a0f90cc91c6812b"} Oct 30 14:48:00.246341 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.246269 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" event={"ID":"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117","Type":"ContainerStarted","Data":"5bc139657334a5a230437f2673fbb86ae124feeeafad141e795e1b529bf941f3"} Oct 30 14:48:00.248077 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.248050 2571 generic.go:356] "Generic (PLEG): container finished" podID="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" containerID="0b5b55dce5a9daeaf938649a055d740b89766384968afa7889fc1a961b574ff6" exitCode=0 Oct 30 14:48:00.248179 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.248134 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbnv" event={"ID":"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076","Type":"ContainerDied","Data":"0b5b55dce5a9daeaf938649a055d740b89766384968afa7889fc1a961b574ff6"} Oct 30 14:48:00.250741 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.250715 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" event={"ID":"0a396a10-b996-4d83-9783-f4a6a543ff6e","Type":"ContainerStarted","Data":"5e46e6043083347b84fb246f1053473ff7292dd2dfefe65f78736e5987c192c0"} Oct 30 14:48:00.253500 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.253473 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d"} Oct 30 14:48:00.362603 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:00.362197 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" podStartSLOduration=2.28329983 podStartE2EDuration="3.362185411s" totalImagesPullingTime="1.078885581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:00.361798508 +0000 UTC m=+56.980328422" watchObservedRunningTime="2026-10-30 14:48:00.362185411 +0000 UTC m=+56.980715325" Oct 30 14:48:01.260152 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:01.260120 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" event={"ID":"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117","Type":"ContainerStarted","Data":"2c83c3c76eeb587c6f715d84630a9241ae3aff70c2a01f1afc129dbcb8eaefa7"} Oct 30 14:48:01.263498 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:01.263413 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbnv" event={"ID":"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076","Type":"ContainerStarted","Data":"082462bcab2217199eb23e2169411f11b399e591e48ef883c2ef4e993b19f07e"} Oct 30 14:48:01.263498 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:01.263465 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jkbnv" event={"ID":"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076","Type":"ContainerStarted","Data":"307b5027a6a85583d4bae868c22ce7d40e9ccff80b1ef0aabfa2077b200d0350"} Oct 30 14:48:01.316179 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:01.316129 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" podStartSLOduration=2.918342292 podStartE2EDuration="4.316110036s" totalImagesPullingTime="1.397767744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:01.314573475 +0000 UTC m=+57.933103390" watchObservedRunningTime="2026-10-30 14:48:01.316110036 +0000 UTC m=+57.934639951" Oct 30 14:48:01.361093 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:01.361045 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jkbnv" podStartSLOduration=2.434960345 podStartE2EDuration="3.361028958s" totalImagesPullingTime="843.419752ms" totalInitContainerRuntime="82.648861ms" isStatefulPod=true podCreationTimestamp="2026-10-30 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:01.36065624 +0000 UTC m=+57.979186155" watchObservedRunningTime="2026-10-30 14:48:01.361028958 +0000 UTC m=+57.979558874" Oct 30 14:48:02.268784 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.268751 2571 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" exitCode=0 Oct 30 14:48:02.269209 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.268836 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} Oct 30 14:48:02.270607 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.270575 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748747647-9rm5d" event={"ID":"88a21bd0-b038-400e-8e85-be20c12f8ec0","Type":"ContainerStarted","Data":"6bda8771326d8afa36c484da6dc5e2d19dba5f92ebaa84f1234855203eb6dc26"} Oct 30 14:48:02.417697 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.417648 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-748747647-9rm5d" podStartSLOduration=1.330477398 podStartE2EDuration="4.417635429s" totalImagesPullingTime="3.087158031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:02.41650867 +0000 UTC m=+59.035038583" watchObservedRunningTime="2026-10-30 14:48:02.417635429 +0000 UTC m=+59.036165340" Oct 30 14:48:02.793463 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.793433 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq"] Oct 30 14:48:02.796592 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.796573 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.838078 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.838053 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 14:48:02.838223 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.838053 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 14:48:02.844535 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.844516 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq"] Oct 30 14:48:02.867277 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867254 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/525049c6-49f7-46f1-8f0e-b31091e03f26-audit-log\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.867466 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867299 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-client-certs\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.867466 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-tls\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.867559 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867493 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg62n\" (UniqueName: \"kubernetes.io/projected/525049c6-49f7-46f1-8f0e-b31091e03f26-kube-api-access-sg62n\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.867559 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867552 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.867634 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867572 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-client-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.867634 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.867618 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-metrics-server-audit-profiles\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.909697 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.909673 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 14:48:02.909826 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.909811 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-881julv4d1pcu\"" Oct 30 14:48:02.909902 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.909886 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9w6js\"" Oct 30 14:48:02.909975 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.909962 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 14:48:02.968336 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968310 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-client-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.968474 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968380 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-metrics-server-audit-profiles\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.968474 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/525049c6-49f7-46f1-8f0e-b31091e03f26-audit-log\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.968474 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968459 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-client-certs\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.968642 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968495 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-tls\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.968642 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg62n\" (UniqueName: \"kubernetes.io/projected/525049c6-49f7-46f1-8f0e-b31091e03f26-kube-api-access-sg62n\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.968642 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.968529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.969137 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.969058 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/525049c6-49f7-46f1-8f0e-b31091e03f26-audit-log\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.969700 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.969676 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-metrics-server-audit-profiles\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.970113 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.970093 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.970913 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.970889 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-client-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.971021 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.970938 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-tls\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.971021 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.970979 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-client-certs\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:02.996156 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:02.996136 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg62n\" (UniqueName: \"kubernetes.io/projected/525049c6-49f7-46f1-8f0e-b31091e03f26-kube-api-access-sg62n\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:03.109901 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.109850 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:03.218897 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.218214 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:03.416770 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.416740 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq"] Oct 30 14:48:03.420081 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.420063 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.444619 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.444595 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 14:48:03.444780 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.444632 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 14:48:03.444780 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.444725 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 14:48:03.444988 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.444966 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lj7tc\"" Oct 30 14:48:03.445067 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.445003 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 14:48:03.445067 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.445010 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 14:48:03.446555 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.446531 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq"] Oct 30 14:48:03.449636 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:03.449610 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525049c6_49f7_46f1_8f0e_b31091e03f26.slice/crio-a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2 WatchSource:0}: Error finding container a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2: Status 404 returned error can't find the container with id a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2 Oct 30 14:48:03.470671 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.470651 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 14:48:03.472456 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.472438 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq"] Oct 30 14:48:03.474153 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474129 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-serving-certs-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474234 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474173 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hlf\" (UniqueName: \"kubernetes.io/projected/5e0152b3-9c26-4191-bfaf-182715417b9b-kube-api-access-48hlf\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474307 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474284 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474369 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474315 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-federate-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474424 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474371 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474424 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474410 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-metrics-client-ca\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474509 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474464 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.474509 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.474501 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575333 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575287 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575491 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575341 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575491 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575418 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-serving-certs-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575491 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575449 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hlf\" (UniqueName: \"kubernetes.io/projected/5e0152b3-9c26-4191-bfaf-182715417b9b-kube-api-access-48hlf\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575646 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575506 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575646 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-federate-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575646 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.575646 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.575593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-metrics-client-ca\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.576370 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.576321 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-metrics-client-ca\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.576608 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.576585 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.577004 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.576942 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-serving-certs-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.578434 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.578409 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.578534 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.578524 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.578812 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.578789 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-federate-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.579012 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.578991 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.600926 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.600899 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hlf\" (UniqueName: \"kubernetes.io/projected/5e0152b3-9c26-4191-bfaf-182715417b9b-kube-api-access-48hlf\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:03.739098 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:03.739040 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:04.094534 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.094506 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq"] Oct 30 14:48:04.109736 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:04.109708 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e0152b3_9c26_4191_bfaf_182715417b9b.slice/crio-52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0 WatchSource:0}: Error finding container 52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0: Status 404 returned error can't find the container with id 52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0 Oct 30 14:48:04.282990 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.282920 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} Oct 30 14:48:04.283111 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.283037 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} Oct 30 14:48:04.283111 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.283078 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} Oct 30 14:48:04.283111 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.283103 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} Oct 30 14:48:04.283259 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.283136 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} Oct 30 14:48:04.284389 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.284334 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" event={"ID":"5e0152b3-9c26-4191-bfaf-182715417b9b","Type":"ContainerStarted","Data":"52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0"} Oct 30 14:48:04.285709 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:04.285673 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" event={"ID":"525049c6-49f7-46f1-8f0e-b31091e03f26","Type":"ContainerStarted","Data":"a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2"} Oct 30 14:48:05.151998 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.151967 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:05.156531 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.156509 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.177509 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177487 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 14:48:05.177649 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177567 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 14:48:05.177738 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177684 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 14:48:05.177803 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177769 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 14:48:05.177803 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177773 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 14:48:05.177892 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177834 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 14:48:05.177892 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177867 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tcq4s\"" Oct 30 14:48:05.177983 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.177913 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ctc47fv49jduf\"" Oct 30 14:48:05.178130 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.178108 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 14:48:05.178130 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.178119 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 14:48:05.178276 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.178177 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 14:48:05.179591 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.179576 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 14:48:05.179652 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.179641 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 14:48:05.179799 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.179788 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 14:48:05.202922 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.202904 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 14:48:05.204113 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.204091 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 14:48:05.205671 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.205642 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:05.289869 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.289838 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290011 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.289887 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290011 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.289980 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290093 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290036 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290138 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290094 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290138 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290118 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290221 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290140 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290259 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290240 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290297 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290273 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290373 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290340 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290432 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290419 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290485 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290449 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290536 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290590 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290548 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290590 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290575 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290687 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290736 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290681 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290736 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290723 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290795 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290752 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.290795 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.290781 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.294267 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.294235 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerStarted","Data":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} Oct 30 14:48:05.296494 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.296472 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" event={"ID":"525049c6-49f7-46f1-8f0e-b31091e03f26","Type":"ContainerStarted","Data":"a60fd77b1f9e57dd66235bbfe0ebf6e51acc98b6c899a8ce7a0cabe258227a76"} Oct 30 14:48:05.383113 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.383065 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.039342171 podStartE2EDuration="6.383052617s" totalImagesPullingTime="3.329739514s" totalInitContainerRuntime="13.970932ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:05.379777119 +0000 UTC m=+61.998307033" watchObservedRunningTime="2026-10-30 14:48:05.383052617 +0000 UTC m=+62.001582530" Oct 30 14:48:05.391400 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.391372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.391902 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.391546 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.391902 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.391585 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.391902 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.391617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.391902 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.391722 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.391902 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.391762 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392208 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392067 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392208 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392112 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392208 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392157 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392208 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392417 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392229 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392417 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392256 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392417 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392264 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392650 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392626 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392949 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392730 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.392949 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.392932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.393436 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.393412 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.393551 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.393529 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.393734 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.393713 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.393884 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.393858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.393958 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.393925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.393958 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.393952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.394057 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.394005 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.394414 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.394388 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.394569 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.394550 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.394729 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.394704 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.394895 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.394870 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.397410 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.398079 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.398481 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.398855 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.398918 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.399403 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.399750 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.399685 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.400894 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.400851 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.401002 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.400983 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.401590 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.401564 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.401697 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.401678 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.402079 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.402038 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.418813 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.418788 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.445526 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.445462 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" podStartSLOduration=2.0569389 podStartE2EDuration="3.445445452s" totalImagesPullingTime="1.388506552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:05.44131207 +0000 UTC m=+62.059841986" watchObservedRunningTime="2026-10-30 14:48:05.445445452 +0000 UTC m=+62.063975369" Oct 30 14:48:05.482486 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.482451 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:05.840095 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:05.840027 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:05.842589 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:05.842564 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36a4793_6b81_439f_b357_cc7e80803eb0.slice/crio-09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2 WatchSource:0}: Error finding container 09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2: Status 404 returned error can't find the container with id 09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2 Oct 30 14:48:06.301709 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.301677 2571 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="3136254ef0afc29d67ec4b720c557c528cd9985a0ba39473588d90fdfd96533d" exitCode=0 Oct 30 14:48:06.302209 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.301757 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"3136254ef0afc29d67ec4b720c557c528cd9985a0ba39473588d90fdfd96533d"} Oct 30 14:48:06.302209 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.301847 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2"} Oct 30 14:48:06.304165 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.304143 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" event={"ID":"5e0152b3-9c26-4191-bfaf-182715417b9b","Type":"ContainerStarted","Data":"e6f46c7dd0993896a7bc7eb3547d96f41f4b5c31d04f0085be836e2af6f8315b"} Oct 30 14:48:06.304270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.304196 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" event={"ID":"5e0152b3-9c26-4191-bfaf-182715417b9b","Type":"ContainerStarted","Data":"fefa53fa7ac589ce18442afdde846ead8cf4b1065ebc897d60028efb6b31195b"} Oct 30 14:48:06.304270 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.304229 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" event={"ID":"5e0152b3-9c26-4191-bfaf-182715417b9b","Type":"ContainerStarted","Data":"f651eff312a1a4eb77ca937d75a9391279298c1dc77dd58e5f472040241248e2"} Oct 30 14:48:06.418485 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.418446 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" podStartSLOduration=1.614484901 podStartE2EDuration="3.418434184s" totalImagesPullingTime="1.803949283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:06.41767141 +0000 UTC m=+63.036201336" watchObservedRunningTime="2026-10-30 14:48:06.418434184 +0000 UTC m=+63.036964097" Oct 30 14:48:06.585213 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.585187 2571 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748747647-9rm5d"] Oct 30 14:48:06.634281 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.634260 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-56cf858bf9-pd4k9"] Oct 30 14:48:06.638784 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.638768 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.672842 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.672789 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56cf858bf9-pd4k9"] Oct 30 14:48:06.706939 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.706914 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707026 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.706944 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707026 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.706983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707112 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.707095 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707181 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.707168 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707224 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.707203 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707256 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.707236 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.707293 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.707278 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.808408 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.808385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.808527 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.808420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.808527 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.808439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.808729 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.808712 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.808791 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.808760 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.808841 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.808790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.809193 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.809166 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.809274 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.809227 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.809274 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.809239 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.809371 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.809295 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.809371 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.809333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.809448 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.809375 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.810047 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.810029 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.811468 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.811445 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.814152 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.814129 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.832914 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.832889 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:06.949867 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:06.949813 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:07.264029 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:07.264003 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56cf858bf9-pd4k9"] Oct 30 14:48:07.266173 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:07.266142 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b69c00_6f7c_4048_aa22_c388091fdec2.slice/crio-e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83 WatchSource:0}: Error finding container e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83: Status 404 returned error can't find the container with id e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83 Oct 30 14:48:07.309395 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:07.309344 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cf858bf9-pd4k9" event={"ID":"80b69c00-6f7c-4048-aa22-c388091fdec2","Type":"ContainerStarted","Data":"e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83"} Oct 30 14:48:08.315599 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.315552 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cf858bf9-pd4k9" event={"ID":"80b69c00-6f7c-4048-aa22-c388091fdec2","Type":"ContainerStarted","Data":"a806e1a938f025816ce8ac207f0f36e0a5641894aa00c7dfc14315bd03a914a5"} Oct 30 14:48:08.344480 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.344451 2571 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56cf858bf9-pd4k9"] Oct 30 14:48:08.360761 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.360703 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-56cf858bf9-pd4k9" podStartSLOduration=2.360690153 podStartE2EDuration="2.360690153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:08.359954177 +0000 UTC m=+64.978484090" watchObservedRunningTime="2026-10-30 14:48:08.360690153 +0000 UTC m=+64.979220066" Oct 30 14:48:08.377426 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.377398 2571 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b5c666f7f-p2trj"] Oct 30 14:48:08.380920 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.380901 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.403059 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.403034 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5c666f7f-p2trj"] Oct 30 14:48:08.470456 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.470431 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:08.526966 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.526939 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527103 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.526979 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527150 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.527101 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527150 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.527134 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527249 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.527167 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527331 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.527307 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527404 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.527372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.527453 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.527414 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.628913 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.628843 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629031 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.628915 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629031 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.628945 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629031 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.628969 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629031 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.628999 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629031 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629024 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629084 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629265 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629133 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629378 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629315 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629787 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629756 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629854 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629885 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.629945 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.629886 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.631975 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.631953 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.632085 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.631981 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.653191 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.653169 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:08.693560 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:08.693539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:09.003111 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.003081 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b5c666f7f-p2trj"] Oct 30 14:48:09.005122 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:09.005095 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda70892a8_ba07_4d35_b23a_6884e1b7d1a5.slice/crio-b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720 WatchSource:0}: Error finding container b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720: Status 404 returned error can't find the container with id b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720 Oct 30 14:48:09.320303 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.320277 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c666f7f-p2trj" event={"ID":"a70892a8-ba07-4d35-b23a-6884e1b7d1a5","Type":"ContainerStarted","Data":"8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f"} Oct 30 14:48:09.320705 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.320373 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c666f7f-p2trj" event={"ID":"a70892a8-ba07-4d35-b23a-6884e1b7d1a5","Type":"ContainerStarted","Data":"b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720"} Oct 30 14:48:09.367543 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.367493 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b5c666f7f-p2trj" podStartSLOduration=1.367474008 podStartE2EDuration="1.367474008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:09.365639217 +0000 UTC m=+65.984169144" watchObservedRunningTime="2026-10-30 14:48:09.367474008 +0000 UTC m=+65.986003922" Oct 30 14:48:09.740383 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.740328 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:09.740516 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.740436 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:09.755084 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.755059 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:48:09.755178 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.755109 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:48:09.763377 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.763334 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:09.763491 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.763464 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:09.841401 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.841374 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:09.855946 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.855923 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:48:09.876441 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.876422 2571 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:48:09.884576 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.884556 2571 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:09.965880 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.965860 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\"" Oct 30 14:48:09.971748 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.971731 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:09.972075 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.972060 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\"" Oct 30 14:48:09.977583 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.977563 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:09.980438 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.980301 2571 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\"" Oct 30 14:48:09.986436 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:09.986245 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:10.283260 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.283129 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cjwh8"] Oct 30 14:48:10.286993 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:10.286952 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4333e0_6941_4815_844e_b7641b1ae720.slice/crio-fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600 WatchSource:0}: Error finding container fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600: Status 404 returned error can't find the container with id fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600 Oct 30 14:48:10.303713 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.303628 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fnbqw"] Oct 30 14:48:10.306134 ip-10-0-139-204 kubenswrapper[2571]: W1030 14:48:10.306106 2571 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f8be0f7_d4c4_4bc9_ae48_ef845d362377.slice/crio-5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec WatchSource:0}: Error finding container 5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec: Status 404 returned error can't find the container with id 5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec Oct 30 14:48:10.314296 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.314053 2571 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bz4fj"] Oct 30 14:48:10.328297 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.328264 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"5c997bb1c8f930ff2dc91bcb785a7a1dc18c9276f49fdf17dd924769a70b359b"} Oct 30 14:48:10.328653 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.328341 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"6dde33c5de4f18713c381b4d5d0a5ecc767d771aee7b723fab08b68d69b8c998"} Oct 30 14:48:10.329651 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.329628 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bz4fj" event={"ID":"1c36de38-25e2-4eb1-9b01-a7c342084792","Type":"ContainerStarted","Data":"5f687b68853e2e396f8c6fcef9b6dad5ab49df7d6fdc025dcbd9f653316ed785"} Oct 30 14:48:10.330738 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.330720 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fnbqw" event={"ID":"4f8be0f7-d4c4-4bc9-ae48-ef845d362377","Type":"ContainerStarted","Data":"5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec"} Oct 30 14:48:10.332741 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:10.332213 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cjwh8" event={"ID":"8d4333e0-6941-4815-844e-b7641b1ae720","Type":"ContainerStarted","Data":"fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600"} Oct 30 14:48:12.345404 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:12.345370 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cjwh8" event={"ID":"8d4333e0-6941-4815-844e-b7641b1ae720","Type":"ContainerStarted","Data":"db0d670344069fa2c89e84f31106b55f6035ee31f693c5717c10cd5f2af13a1f"} Oct 30 14:48:13.351923 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.351890 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cjwh8" event={"ID":"8d4333e0-6941-4815-844e-b7641b1ae720","Type":"ContainerStarted","Data":"252082d6f83a10b737b9c42448b64994d0eeb4393bd3e811f6704fedef3d743d"} Oct 30 14:48:13.356438 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.356409 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"4fdaf243b2ba0e72e8e2210480f797faf17330681e8c0635db7a732609af7c9c"} Oct 30 14:48:13.356577 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.356481 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"69fa3f707c2c52ddca4de1c012377a567ad42ba2eb1bf5c3345a34d41538e3f0"} Oct 30 14:48:13.356577 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.356526 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"5aa58ce3fee9d2dfbcbd8664ffd0ec3e1fac15d84b1669fcaf3880dd610e84ee"} Oct 30 14:48:13.356577 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.356571 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerStarted","Data":"c1aae2d5db9089161023e0f1e98576bb74657274a17def109ac078cb7af80a8a"} Oct 30 14:48:13.394910 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.394865 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cjwh8" podStartSLOduration=66.489977958 podStartE2EDuration="1m8.394851021s" totalImagesPullingTime="1.904873063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:13.393909463 +0000 UTC m=+70.012439377" watchObservedRunningTime="2026-10-30 14:48:13.394851021 +0000 UTC m=+70.013380938" Oct 30 14:48:13.464305 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:13.464253 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.347191505 podStartE2EDuration="8.46423928s" totalImagesPullingTime="6.097802148s" totalInitContainerRuntime="19.245627ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:13.463198292 +0000 UTC m=+70.081728242" watchObservedRunningTime="2026-10-30 14:48:13.46423928 +0000 UTC m=+70.082769194" Oct 30 14:48:15.368176 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:15.368131 2571 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fnbqw" event={"ID":"4f8be0f7-d4c4-4bc9-ae48-ef845d362377","Type":"ContainerStarted","Data":"fc1606dd10252562f8baf6ef78d14cb83f449e5c689dadddb984d1a1992dc363"} Oct 30 14:48:15.368630 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:15.368259 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:15.413941 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:15.413880 2571 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fnbqw" podStartSLOduration=66.414130652 podStartE2EDuration="1m10.413866485s" totalImagesPullingTime="3.999735833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:15.411239792 +0000 UTC m=+72.029769705" watchObservedRunningTime="2026-10-30 14:48:15.413866485 +0000 UTC m=+72.032396401" Oct 30 14:48:15.482865 ip-10-0-139-204 kubenswrapper[2571]: I1030 14:48:15.482835 2571 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:15.976668 ip-10-0-139-204 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 14:48:15.992794 ip-10-0-139-204 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 14:48:15.992971 ip-10-0-139-204 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 14:48:15.993021 ip-10-0-139-204 systemd[1]: kubelet.service: Consumed 4.265s CPU time. Oct 30 14:48:16.008942 ip-10-0-139-204 systemd[1]: Starting Kubernetes Kubelet... Oct 30 14:48:16.068540 ip-10-0-139-204 kubenswrapper[6554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:48:16.068540 ip-10-0-139-204 kubenswrapper[6554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 14:48:16.068540 ip-10-0-139-204 kubenswrapper[6554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:48:16.069037 ip-10-0-139-204 kubenswrapper[6554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 14:48:16.070553 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070482 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.070553 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070552 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070557 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070562 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070566 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070569 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070572 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070575 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070577 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070580 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070583 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070585 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070588 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070591 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070593 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070596 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070599 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070601 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070604 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070609 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.070630 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070613 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070616 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070618 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070622 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070625 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070628 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070631 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070641 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070644 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070647 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070650 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070652 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070655 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070658 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070661 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070663 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070666 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070668 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070671 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070674 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.071080 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070677 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070679 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070683 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070687 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070690 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070692 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070695 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070697 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070700 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070703 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070705 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070708 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070710 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070714 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070718 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070721 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070724 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070726 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070729 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.071584 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070732 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070735 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070737 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070740 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070743 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070745 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070748 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070750 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070753 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070755 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070758 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070760 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070782 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070786 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070789 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070792 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070794 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070798 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070801 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070803 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.072047 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070806 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070809 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070811 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070814 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070816 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070819 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070822 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070825 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070828 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070831 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070833 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070836 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070839 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070841 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070844 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070847 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070849 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070852 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070855 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.072529 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070857 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070860 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070864 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070866 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070869 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070871 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070874 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.070877 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071626 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071633 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071636 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071639 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071642 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071645 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071648 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071651 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071653 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071656 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071659 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071662 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.072978 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071665 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071669 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071674 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071678 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071681 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071684 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071686 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071689 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071692 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071695 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071698 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071700 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071703 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071707 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071709 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071712 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071715 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071717 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071720 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.073470 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071722 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071725 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071727 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071730 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071732 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071735 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071738 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071740 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071743 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071746 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071749 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071752 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071755 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071757 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071760 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071762 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071765 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071767 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071770 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071773 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.073944 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071775 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071778 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071783 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071786 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071789 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071791 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071794 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071797 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071799 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071802 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071805 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071807 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071810 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071812 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071815 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071817 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071820 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071823 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071825 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.074486 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071828 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071830 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071833 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071836 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071840 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071844 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071847 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071849 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071852 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071855 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071857 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071860 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071862 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071865 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071868 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071870 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071873 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071876 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071878 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.074939 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071882 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071884 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071887 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071889 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071892 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071894 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071897 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071900 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071902 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071905 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071907 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071910 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071912 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071915 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071918 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071921 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.071924 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072417 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072423 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.075401 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072425 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072428 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072431 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072434 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072436 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072439 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072441 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072444 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072447 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072449 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072453 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072455 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072458 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072461 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072463 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072466 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072469 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072471 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072474 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.075849 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072477 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072481 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072485 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072488 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072492 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072495 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072499 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072501 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072504 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072509 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072512 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072516 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072518 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072521 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072524 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072527 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072529 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072532 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072535 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.076312 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072538 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072541 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072543 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072546 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072548 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072551 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072554 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072557 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072560 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072562 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072565 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072567 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072570 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072573 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072575 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072578 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072581 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072583 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072586 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072588 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.076780 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072591 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072594 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072596 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072599 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072601 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072604 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072606 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072609 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072611 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072614 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072616 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072619 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072621 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072624 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072627 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072630 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072632 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072635 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072638 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.077258 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072641 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072644 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072646 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072649 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072652 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072654 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072657 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072659 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072662 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072665 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072667 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072671 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072673 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072676 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072678 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072681 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072683 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072685 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072688 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.077921 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072690 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072693 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072695 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072698 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072700 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072703 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072705 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.072708 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072807 6554 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072825 6554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072835 6554 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072843 6554 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072847 6554 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072851 6554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072856 6554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072860 6554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072863 6554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072867 6554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072870 6554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072873 6554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072876 6554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072879 6554 flags.go:64] FLAG: --cgroup-root="" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072882 6554 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 14:48:16.078664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072885 6554 flags.go:64] FLAG: --client-ca-file="" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072889 6554 flags.go:64] FLAG: --cloud-provider="external" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072892 6554 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072896 6554 flags.go:64] FLAG: --cluster-domain="" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072898 6554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072901 6554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072904 6554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072908 6554 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072911 6554 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072914 6554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072917 6554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072920 6554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072923 6554 flags.go:64] FLAG: --contention-profiling="false" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072926 6554 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072929 6554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072933 6554 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072935 6554 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072939 6554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072942 6554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072945 6554 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072949 6554 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072952 6554 flags.go:64] FLAG: --enable-server="true" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072955 6554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072962 6554 flags.go:64] FLAG: --event-burst="100" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072965 6554 flags.go:64] FLAG: --event-qps="50" Oct 30 14:48:16.079343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072968 6554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072971 6554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072974 6554 flags.go:64] FLAG: --eviction-hard="" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072978 6554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072982 6554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072985 6554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072988 6554 flags.go:64] FLAG: --eviction-soft="" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072991 6554 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072994 6554 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.072997 6554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073000 6554 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073003 6554 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073006 6554 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073009 6554 flags.go:64] FLAG: --feature-gates="" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073013 6554 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073016 6554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073019 6554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073023 6554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073026 6554 flags.go:64] FLAG: --healthz-port="10248" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073029 6554 flags.go:64] FLAG: --help="false" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073032 6554 flags.go:64] FLAG: --hostname-override="ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073035 6554 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073038 6554 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 14:48:16.080002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073041 6554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073044 6554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073048 6554 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073051 6554 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073055 6554 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073058 6554 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073061 6554 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073069 6554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073072 6554 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073075 6554 flags.go:64] FLAG: --kube-reserved="" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073078 6554 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073081 6554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073084 6554 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073087 6554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073090 6554 flags.go:64] FLAG: --lock-file="" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073093 6554 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073096 6554 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073099 6554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073104 6554 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073107 6554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073110 6554 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073112 6554 flags.go:64] FLAG: --logging-format="text" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073115 6554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073119 6554 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 14:48:16.080563 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073121 6554 flags.go:64] FLAG: --manifest-url="" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073124 6554 flags.go:64] FLAG: --manifest-url-header="" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073129 6554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073132 6554 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073135 6554 flags.go:64] FLAG: --max-pods="110" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073138 6554 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073141 6554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073144 6554 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073147 6554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073150 6554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073153 6554 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073156 6554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073164 6554 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073167 6554 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073170 6554 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073174 6554 flags.go:64] FLAG: --pod-cidr="" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073177 6554 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073184 6554 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073187 6554 flags.go:64] FLAG: --pods-per-core="0" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073190 6554 flags.go:64] FLAG: --port="10250" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073193 6554 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073196 6554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05440f16bb9cda115" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073199 6554 flags.go:64] FLAG: --qos-reserved="" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073202 6554 flags.go:64] FLAG: --read-only-port="10255" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073205 6554 flags.go:64] FLAG: --register-node="true" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073208 6554 flags.go:64] FLAG: --register-with-taints="" Oct 30 14:48:16.081134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073212 6554 flags.go:64] FLAG: --registry-burst="10" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073215 6554 flags.go:64] FLAG: --registry-qps="5" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073218 6554 flags.go:64] FLAG: --reserved-cpus="" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073221 6554 flags.go:64] FLAG: --reserved-memory="" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073225 6554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073228 6554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073231 6554 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073234 6554 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073236 6554 flags.go:64] FLAG: --runonce="false" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073240 6554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073243 6554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073246 6554 flags.go:64] FLAG: --seccomp-default="false" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073249 6554 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073251 6554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073254 6554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073258 6554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073260 6554 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073263 6554 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073268 6554 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073271 6554 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073274 6554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073278 6554 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073281 6554 flags.go:64] FLAG: --system-cgroups="" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073284 6554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073290 6554 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 14:48:16.081836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073293 6554 flags.go:64] FLAG: --tls-cert-file="" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073296 6554 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073300 6554 flags.go:64] FLAG: --tls-min-version="" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073303 6554 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073305 6554 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073308 6554 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073311 6554 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073314 6554 flags.go:64] FLAG: --v="2" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073318 6554 flags.go:64] FLAG: --version="false" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073322 6554 flags.go:64] FLAG: --vmodule="" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073326 6554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073329 6554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073481 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073485 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073488 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073491 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073494 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073498 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073500 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073503 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073506 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073509 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.082494 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073511 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073514 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073517 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073521 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073524 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073526 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073538 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073542 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073544 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073547 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073550 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073553 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073556 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073559 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073561 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073564 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073566 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073569 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073572 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.083038 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073576 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073580 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073583 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073586 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073589 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073591 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073594 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073597 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073599 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073602 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073604 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073607 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073609 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073613 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073615 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073618 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073622 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073625 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073628 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.083726 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073631 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073634 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073637 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073639 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073642 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073645 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073648 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073650 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073653 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073656 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073659 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073661 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073664 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073666 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073669 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073671 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073674 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073676 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073679 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073681 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.084229 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073684 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073686 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073689 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073692 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073694 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073697 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073700 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073702 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073705 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073709 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073711 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073714 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073718 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073720 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073723 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073726 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073729 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073731 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073734 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073737 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.084802 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073739 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073742 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073744 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073747 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073750 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073752 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073755 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073757 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073760 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073762 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073765 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073767 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073770 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073773 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073775 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073778 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073781 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.085332 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073785 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.073810 6554 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073845 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073849 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073853 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073857 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073859 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073862 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073866 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073868 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073871 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073873 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073876 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073879 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073881 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073884 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.085919 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073887 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073889 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073892 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073894 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073897 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073899 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073902 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073905 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073907 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073910 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073912 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073915 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073918 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073921 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073923 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073927 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073931 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073935 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073938 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073941 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.086426 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073945 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073948 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073951 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073954 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073957 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073960 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073963 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073965 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073968 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073971 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073973 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073976 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073979 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073982 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073984 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073987 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073990 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073992 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073995 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.073998 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.087066 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074000 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074003 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074005 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074008 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074011 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074014 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074017 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074020 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074023 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074026 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074029 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074031 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074034 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074037 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074039 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074042 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074046 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074049 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074051 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.087671 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074054 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074056 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074059 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074062 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074064 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074067 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074070 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074072 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074075 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074077 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074080 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074082 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074085 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074087 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074090 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074093 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074095 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074098 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074100 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.088166 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074103 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074105 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074108 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074111 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074113 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074116 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074118 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074121 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074124 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074126 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074130 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074134 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074136 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.074139 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.079032 6554 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.079048 6554 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.079071 6554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.079076 6554 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079133 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.088632 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079138 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079141 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079144 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079147 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079150 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079153 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079156 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079159 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079162 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079164 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079168 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079170 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079173 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079176 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079179 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079182 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079184 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079187 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079189 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.089082 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079192 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079195 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079197 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079200 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079202 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079205 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079207 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079210 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079213 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079215 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079218 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079221 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079223 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079226 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079228 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079231 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079234 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079236 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079239 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.089551 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079241 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079244 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079246 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079249 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079251 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079254 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079258 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079268 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079271 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079273 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079276 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079279 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079281 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079284 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079287 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079289 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079292 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079294 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079297 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.089998 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079301 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079305 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079307 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079310 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079312 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079315 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079318 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079320 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079323 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079325 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079328 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079330 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079333 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079335 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079338 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079341 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079344 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079362 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079365 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079368 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.090656 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079373 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079383 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079386 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079389 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079392 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079394 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079397 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079400 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079402 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079405 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079407 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079410 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079413 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079415 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079418 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079420 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079423 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079426 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079428 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079430 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.091124 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079433 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079435 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079438 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079440 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079443 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079445 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079448 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079451 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.079479 6554 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079614 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079619 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079623 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079625 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079628 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079636 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079639 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.091603 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079641 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079644 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079647 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079650 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079653 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079655 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079658 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079660 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079663 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079665 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079668 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079671 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079673 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079676 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079678 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079680 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079683 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079685 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079688 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079690 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.092053 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079693 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079695 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079698 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079700 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079704 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079706 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079709 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079712 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079714 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079717 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079719 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079728 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079732 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079736 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079739 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079741 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079744 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079746 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079749 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.092572 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079752 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079754 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079757 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079759 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079764 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079767 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079769 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079772 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079775 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079777 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079780 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079782 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079785 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079788 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079791 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079793 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079796 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079799 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079801 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.093027 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079804 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079806 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079808 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079811 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079814 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079816 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079823 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079826 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079829 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079831 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079834 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079836 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079840 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079843 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079846 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079849 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079852 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079857 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079859 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.093501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079862 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079864 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079867 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079870 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079872 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079875 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079878 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079880 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079883 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079885 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079888 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079891 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079893 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079896 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079899 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079901 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079904 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079906 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079909 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079911 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.094005 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079918 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079921 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.079945 6554 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079985 6554 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079990 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079993 6554 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079996 6554 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.079998 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080001 6554 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080004 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080008 6554 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080011 6554 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080013 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080016 6554 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080019 6554 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 14:48:16.094818 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080022 6554 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080025 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080027 6554 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080030 6554 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080033 6554 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080035 6554 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080038 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080041 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080043 6554 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080046 6554 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080048 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080051 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080053 6554 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080056 6554 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080058 6554 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080061 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080063 6554 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080066 6554 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080073 6554 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 14:48:16.095447 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080076 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080078 6554 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080083 6554 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080086 6554 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080088 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080091 6554 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080093 6554 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080096 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080100 6554 feature_gate.go:386] unrecognized feature gate: Example Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080105 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080107 6554 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080110 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080113 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080115 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080118 6554 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080121 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080123 6554 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080126 6554 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080128 6554 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080131 6554 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 14:48:16.096035 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080133 6554 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080136 6554 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080138 6554 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080141 6554 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080143 6554 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080146 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080149 6554 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080151 6554 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080154 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080156 6554 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080159 6554 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080161 6554 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080169 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080171 6554 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080174 6554 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080177 6554 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080181 6554 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080185 6554 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080189 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080192 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 14:48:16.096544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080195 6554 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080199 6554 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080202 6554 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080204 6554 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080207 6554 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080209 6554 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080212 6554 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080215 6554 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080218 6554 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080220 6554 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080223 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080226 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080228 6554 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080231 6554 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080233 6554 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080236 6554 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080238 6554 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080241 6554 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080244 6554 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 14:48:16.097191 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080247 6554 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080249 6554 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080252 6554 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080254 6554 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080257 6554 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080259 6554 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080266 6554 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080269 6554 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080272 6554 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080275 6554 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080278 6554 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080281 6554 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080283 6554 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080287 6554 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080291 6554 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:16.080294 6554 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.080362 6554 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081265 6554 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081314 6554 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 14:48:16.097782 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081527 6554 server.go:1014] "Starting client certificate rotation" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081664 6554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081704 6554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 14:42:04 +0000 UTC" deadline="2028-08-09 04:04:31.557637734 +0000 UTC" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081730 6554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15565h16m15.475909145s" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.081968 6554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.083080 6554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.084291 6554 log.go:25] "Validated CRI v1 runtime API" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.086501 6554 log.go:25] "Validated CRI v1 image API" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.087288 6554 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 14:48:16.098289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.089620 6554 fs.go:135] Filesystem UUIDs: map[65ed9b72-b0a0-483d-849e-70ebf51a4b8e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9acda3fc-1fa8-49c2-b729-64eac504437c:/dev/nvme0n1p4] Oct 30 14:48:16.098614 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.089638 6554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2/userdata/shm major:0 minor:406 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d/userdata/shm major:0 minor:302 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0/userdata/shm major:0 minor:353 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088/userdata/shm major:0 minor:266 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec/userdata/shm major:0 minor:468 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f687b68853e2e396f8c6fcef9b6dad5ab49df7d6fdc025dcbd9f653316ed785/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f687b68853e2e396f8c6fcef9b6dad5ab49df7d6fdc025dcbd9f653316ed785/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c73bcc3e26c17d123280dceb14916054cdd09f5f3af9f2bd0d2e9e9d0b60c5e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c73bcc3e26c17d123280dceb14916054cdd09f5f3af9f2bd0d2e9e9d0b60c5e6/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d/userdata/shm major:0 minor:262 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e522e488059a8c2d0ba51f94605e25406cff59318ee036f1e13e3c9414b11de6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e522e488059a8c2d0ba51f94605e25406cff59318ee036f1e13e3c9414b11de6/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83/userdata/shm major:0 minor:429 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600/userdata/shm major:0 minor:466 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~projected/kube-api-access-rhbcn:{mountpoint:/var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~projected/kube-api-access-rhbcn major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/volumes/kubernetes.io~projected/kube-api-access-cp8wt:{mountpoint:/var/lib/kubelet/pods/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/volumes/kubernetes.io~projected/kube-api-access-cp8wt major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c36de38-25e2-4eb1-9b01-a7c342084792/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/1c36de38-25e2-4eb1-9b01-a7c342084792/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50/volumes/kubernetes.io~projected/kube-api-access-5hp9b:{mountpoint:/var/lib/kubelet/pods/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50/volumes/kubernetes.io~projected/kube-api-access-5hp9b major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b36b9d5-51b0-4038-ae52-a7d1299cd284/volumes/kubernetes.io~projected/kube-api-access-7mgmz:{mountpoint:/var/lib/kubelet/pods/2b36b9d5-51b0-4038-ae52-a7d1299cd284/volumes/kubernetes.io~projected/kube-api-access-7mgmz major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b36b9d5-51b0-4038-ae52-a7d1299cd284/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/2b36b9d5-51b0-4038-ae52-a7d1299cd284/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~projected/kube-api-access-h2k5h:{mountpoint:/var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~projected/kube-api-access-h2k5h major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/alertmanager/9 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~empty-dir/config-out major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~projected/kube-api-access-6pj85:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~projected/kube-api-access-6pj85 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~projected/tls-assets major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/config-volume major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/web-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f8be0f7-d4c4-4bc9-ae48-ef845d362377/volumes/kubernetes.io~projected/kube-api-access-7nzhv:{mountpoint:/var/lib/kubelet/pods/4f8be0f7-d4c4-4bc9-ae48-ef845d362377/volumes/kubernetes.io~projected/kube-api-access-7nzhv major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~projected/kube-api-access-sg62n:{mountpoint:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~projected/kube-api-access-sg62n major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a72a542-4dd5-46f1-bdba-e82aba26d948/volumes/kubernetes.io~projected/kube-api-access-87sz4:{mountpoint:/var/lib/kubelet/pods/5a72a542-4dd5-46f1-bdba-e82aba26d948/volumes/kubernetes.io~projected/kube-api-access-87sz4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~projected/kube-api-access-48hlf:{mountpoint:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~projected/kube-api-access-48hlf major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d201227-679c-4342-8bbc-f386d540cc81/volumes/kubernetes.io~projected/kube-api-access-mslsf:{mountpoint:/var/lib/kubelet/pods/6d201227-679c-4342-8bbc-f386d540cc81/volumes/kubernetes.io~projected/kube-api-access-mslsf major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~projected/kube-api-access-kgrpk:{mountpoint:/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~projected/kube-api-access-kgrpk major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:423 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~projected/kube-api-access-df7bw:{mountpoint:/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~projected/kube-api-access-df7bw major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d4333e0-6941-4815-844e-b7641b1ae720/volumes/kubernetes.io~projected/kube-api-access-qrksh:{mountpoint:/var/lib/kubelet/pods/8d4333e0-6941-4815-844e-b7641b1ae720/volumes/kubernetes.io~projected/kube-api-access-qrksh major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d4333e0-6941-4815-844e-b7641b1ae720/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/8d4333e0-6941-4815-844e-b7641b1ae720/volumes/kubernetes.io~secret/metrics-certs major:0 minor:464 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e5f527d-4282-4ace-b004-5a2504a1d73e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8e5f527d-4282-4ace-b004-5a2504a1d73e/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~projected/kube-api-access-rmqf7:{mountpoint:/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~projected/kube-api-access-rmqf7 major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa997701-363d-4c89-98f7-b6e5c349b892/volumes/kubernetes.io~projected/kube-api-access-48bzs:{mountpoint:/var/lib/kubelet/pods/aa997701-363d-4c89-98f7-b6e5c349b892/volumes/kubernetes.io~projected/kube-api-access-48bzs major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2e9a5db-d5d7-4b26-828a-203c6557820e/volumes/kubernetes.io~projected/kube-api-access-bcfzz:{mountpoint:/var/lib/kubelet/pods/d2e9a5db-d5d7-4b26-828a-203c6557820e/volumes/kubernetes.io~projected/kube-api-access-bcfzz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da158b42-504b-4dfa-9b41-3333f4bbf0e7/volumes/kubernetes.io~projected/kube-api-access-n9xx9:{mountpoint:/var/lib/kubelet/pods/da158b42-504b-4dfa-9b41-3333f4bbf0e7/volumes/kubernetes.io~projected/kube-api-access-n9xx9 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~projected/kube-api-access-prsvj:{mountpoint:/var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~projected/kube-api-access-prsvj major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/config-reloader/2 major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/prometheus/15 major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~empty-dir/config-out major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~projected/kube-api-access-4kpnl:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~projected/kube-api-access-4kpnl major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~projected/tls-assets major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/config major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/web-config major:0 minor:397 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~projected/kube-api-access-b9lcg:{mountpoint:/var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~projected/kube-api-access-b9lcg major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:256 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ee24becd-13c7-4fd5-a314-0b5f27a066ad/volumes/kubernetes.io~projected/kube-api-access-mfq98:{mountpoint:/var/lib/kubelet/pods/ee24becd-13c7-4fd5-a314-0b5f27a066ad/volumes/kubernetes.io~projected/kube-api-access-mfq98 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volumes/kubernetes.io~projected/kube-api-access-4xcnn:{mountpoint:/var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volumes/kubernetes.io~projected/kube-api-access-4xcnn major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/10d94d534be8e05fe31c55f26bb52ba7247732ab5a1d34a4b6ce2fb76fb70d28/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/502654af67725a1c8a8057d095e9f0ea9b0e99fa64abb89fda1dc5eb3cdb38ae/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/51b7d3e3e714b765c97ceef8ff0689ad6b5a39454013d0e86c4b5ec99d5415f9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-105:{mountpoint:/var/lib/containers/storage/overlay/d85304af7c386d1c1d3e79a3582dac2bd4dc8689ad3f0d42fa5c8e1684c10a34/merged major:0 minor:105 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/afc8a1cdc6312edcdc280ad24e5953711e89783f407db8f10c95fb5b6c8c639c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/441d84aaf06de03f676829d941d9a3224ef1d83359003fc8d80e680af979016a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/d666c5160e3f6ecd4a68cff1f77d5c1637c6b38b2e062a7f532ef5e9dccbd7f3/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ec29309ff1be10eb4fc50221fc69d2b2bad86fc18ad9baf5140d4e481821f934/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e8cd8bdb4f60f22e2122f216ef2ca9aabb1f4dbe3c2a83659a86fca3387bde4a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/3117fc97d04eb7a5401c8ebf889562cc5c1491803f88a5ecbf0ede2ce7d36784/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/533a0b4b4e9b957c4596ab6fd702f5fb46156253891e37fed5ffaacfffcd240e/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/34970dfa93bd98d94aab1cfb091900d36bc4b4817d3c29694e4d8163dd66656a/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/92b9912e7eb42ac277f1ede18b0051218139b5f088f0b42bda86bbb19134e664/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/4f614b68aeed81d6e48591cb0b8e793dcd9a2ff62d81b7e2981caac69a202007/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/549854ebf11154af7db3ee56817359bfea80fa75037376ade8cb90b1c6a49617/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/8d08e8e076ecdcd878dc423729527d39751340fa37c3f2377eca9052cc4f9af1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/03a3a6974f7511d159ccf8d71b7b6a3d95e224124f4b8e2b621bcd2146a41daa/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/906720a16a055cb5afae59537a7e831e0b7e684bae0f1e03085cc849f4798e35/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/ac8fcce990a70b9e96187366e02f6ecbe1745494932c8c14164041579831302a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/001b22d49fb1bf45a220ebf4b70dfdfe2f0303115a0b976c91a79e94d6890ccb/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8783990e9d26e38a60446b46c0cb94de923ddee05fa9cd9ab4876e9a09c97f86/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/cb83cdae1bb5a7d672ade0256ea16980bb14d83a060773a39085e77986dee42a/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/f26a4507c66f0d403004007a1b6da31b1f232b6eddf544f57e6de41f20596bae/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f4802a56c35e2e9f9e1cc2f3b8ecf1346cd675151fa37c4ead602e65795bedcd/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/164c1298a82775b8ecc1161a26d08223106e4437c2ed65a3a2f495456f7526a7/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/81ad6d5f0b02cfdc2339224693db25362715c5d014f66ce9df6af12dd2640d61/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/f4ce55c37a6923aaca5d73f117d83adcb1bbf58d4029b677d2cb65a680d5bc6d/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/34d12d05a6f4e8f917bf9230a5b23f645b033d29bf9ee08e99ca9f06be61e7de/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/e40124c9ff9a03e6fbdf681f6c5b47afb292c58d8e73ba5ad5d22039d825c87e/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/debb65106c46c3d5fb9b4fc09da70350eccb4126fa2903e916e74fe885a56df6/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/4bb6eb5101aadd7c786bd6711633e4ec51335d683d7b669ec96a5356f62da06a/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/3fca21e50051b167715fda6194f9827242ff64d64f3270a5dc4c44d46dea0576/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/8d1ad025aa5a24300ddb9135276fa7d96c8f7ed22e71c88ad89a469a3b03022d/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/7a2135443f2d6ec78189e0e118074136dcf791f224215921ccc1dfc8848eb9ef/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/69b13ccd47898547b16b68e17fec78a334cf58cbdb74f5b7812fb86825a73242/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/ff02c988fdeff6c5d0a69cc5149249d794e9371173b32d41475aaa486d9027ba/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/742678ba17efc3ccb6f01bbbc3a2aa69802d0f3561d08685c7dc18c53a6d48c8/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/d2808727052bfcd8b441ea170340d3dc24ea04f2193484f095d4da4bc691df83/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/22159580ce23099620cde14ed9188900b1d77aaf98d8f23f2c36d5d441e05216/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/2e6bccbd9025fe7397fb702475d625148cff24d3f96a93c7b697a103a3245db7/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/157940db9c82f7cb78155321e644234815002b2aa7a872c5959428bc7e34e851/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/49b79505967ef53344bc3cbd54f919922e7f2252ffa71da55fbc880f9d085f54/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/15fdc3080c8c01a2087515ff32915f7817b9f9965341726aa7a944f08f1010b1/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/9353193101b379427243d4b5d25edf697439d41bdb421699582663b27960a988/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/f7fd8ca3cb9db23c78524d31537ee1fcde4a1f5a0511dbfaea6684ce547f5059/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/d10a2ccacaff03daa3e52b389adffd71abdfd6f29f7c64a38d7cc4d08dc0c47d/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/4c2f593fc448135db1f4bdde4326dc94db418af08fd60fa5d8a3c48444c38b1b/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/698cfbe04ed020a2b52f7f2811070a6a99d81db96d1685bac3a5351c4557955c/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/e135877f3a62ddcdae316310119135ebddfce5ce31d1ffb232719079f671dcf6/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-373:{mountpoint:/var/lib/containers/storage/overlay/c4bf54ee3b14c35d9cf1865046b6346beabd9c1d64fb32bd86243fdea607f8cf/merged major:0 minor:373 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/0a55b9d0b4575dcf2c6765157b95b2a2833d65bc8173a3102cea464908622b01/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/46b3a811d86cc453837aa0dc153e303c64adce2841f2fcc06c6ab5d7df064e91/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/eda315a364af2ce583cebac0ade4c7aa09f9176ce7861c5c1c1e940424994cb7/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/7c84d7ec39fb615641eff6b2283bee09f5fcf948522cb9fc163d27c40127bb64/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/bd6840fe77aa16a68b96af9e5d746dce9152c104672a962929a81d461e2c1fc4/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/aaceea9539fa2e84ee838b7fe4b40243eb41c4fc97e797d45b5625f2e57576ac/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-418:{mountpoint:/var/lib/containers/storage/overlay/a8b786eafe67c041811692b4fa813657d6aae8c58ef1c57b71eb99ff5d097ff7/merged major:0 minor:418 fsType:overlay blockSize:0} overlay_0-431:{mountpoint:/var/lib/containers/storage/overlay/5bcabec22f1c992170526d54013656b080b41f05035bd25419f78368faefb43e/merged major:0 minor:431 fsType:overlay blockSize:0} overlay_0-433:{mountpoint:/var/lib/containers/storage/overlay/e237e03c53c35fba6c78e377beef46f38b23795c09ffd716dee1c008c7ec82ac/merged major:0 minor:433 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/eb2d7ce45ee2e694348ea6d4394bcfd9be30124879c0c7ffc61f2a0909d787f8/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/30e7a9bfa2c1f9b788a54c7e724e3afae3c7a8dae60e35c9ed296c26f34f4a84/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-448:{mountpoint:/var/lib/containers/storage/overlay/9af2ced033d6eba20fc02c2cb9d2d30ada3e5641184abf055fd4b23f55c00fca/merged major:0 minor:448 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/bf2b2cbc33115aff014e05af3d5be3e9ad265565bdf681938fe420e21754148a/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/f39d6384fd8f7ff22f509389f090dbd31bac888b484e48d49169ca0ad4eea1ef/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/8a9f96ce830a1ea499202764fd13c82c0f42797efa74e0c6d04da227d3b47109/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/6b785a8e3c8e0aa04bebee1541665d2a94b218b3c7891a288374bf1e35efd6bb/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/d768eb11d0ad399c14199c3dee157d994995444d8e843dc300b341ee74df8646/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/2e35ea40650cc93454aad9a1dea28e9acf21b89b800fd1f86a3b4a1bd71e9774/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-486:{mountpoint:/var/lib/containers/storage/overlay/6b9000ab7e6953e64effa3b77c656afdb3a1cc137a0f044f34ce989ff95cd696/merged major:0 minor:486 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/a131721b386b8975a5d5fc7c53a3c36ee7ba1d328d665af7d922375e94f6e112/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-491:{mountpoint:/var/lib/containers/storage/overlay/3388a14065a80dbeef84ee081982b1b8cc7095f6a6192ebb8849da562aac8d13/merged major:0 minor:491 fsType:overlay blockSize:0} overlay_0-496:{mountpoint:/var/lib/containers/storage/overlay/ac21a780466d63493d58b4049d37a902c0e253725d87a134cf1f78c4cd526035/merged major:0 minor:496 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/a6bdcbadfdaa92092bb2950cf723debf2cdf35351d6e0084ba37a1cf3e29371d/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/099b376d6a0ec313350ead958ad91922d3c0b14756d4f2c2afe3fac4aa30be49/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-511:{mountpoint:/var/lib/containers/storage/overlay/e30c38be4a030b038f7cf965479bb2985e2b5aa75e2b56e0cd5dfe48c4246693/merged major:0 minor:511 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/afb5aee58b5b980fa52cd62732a65440c28f56374c2e6e85664ce0255ced94f6/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b8239a823995e832cfe887ee3adcd85e98a3738e71bb7726ad87e00c498f107a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/3f285b27ce9e78483fd5bb723dd76d3a3e3004c33ecbaea4d8e4f254f79bf41f/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1210198a495cfcc69358fc1484dbbd499dd44b637edf8cc7d86008f9e6d10567/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/f828a91829be5483690c08087bcb2c673123370fcd44f21c256866e5dfb69550/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d4f9000ee7d0811eb42b277a6f40add43e76a88804740150081fe013f6d1df69/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1b49474059518563a610f82a0f294c2f462b5b9dc3450644a7cea643b666ceee/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/02a322f91040e195513829e7b5db75bff77dc17e7eda850bef675f8b5f68aab8/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d5e72768f8c58652716c2c41ee9f5423fce6000472ee6e931798ddfed8d3d4f2/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 14:48:16.099118 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.098674 6554 manager.go:217] Machine: {Timestamp:2026-10-30 14:48:16.098199762 +0000 UTC m=+0.057241157 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100177 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20fe6dbad8527ab1d85b8978ee61ad SystemUUID:ec20fe6d-bad8-527a-b1d8-5b8978ee61ad BootID:1aceed49-55ae-47f4-85c9-74906c8da00c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:404 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~projected/kube-api-access-b9lcg DeviceMajor:0 DeviceMinor:257 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~projected/kube-api-access-kgrpk DeviceMajor:0 DeviceMinor:428 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ee24becd-13c7-4fd5-a314-0b5f27a066ad/volumes/kubernetes.io~projected/kube-api-access-mfq98 DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b28e778765e389c6d3f59cf44ae7325795fa35cfdede1fd429e49d3c8f369ec/userdata/shm DeviceMajor:0 DeviceMinor:468 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-105 DeviceMajor:0 DeviceMinor:105 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a496ebc2ccba3189eec3b686ee0f43fe4ed058d6af016172b41077114614bae2/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:395 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:403 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d/userdata/shm DeviceMajor:0 DeviceMinor:302 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~projected/kube-api-access-rmqf7 DeviceMajor:0 DeviceMinor:441 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbc64dc6a549cbcca926fcef9df9f79f694a2b4f532ad869f57e78a2243fd600/userdata/shm DeviceMajor:0 DeviceMinor:466 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af44c877a73ce95e2498db50acca9105a8b07a3c50c976b1646462f26892c586/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df6ee019439ff0eb00342a3cdeb4ec12c40466f059a5301f981c30bc15fb4d4d/userdata/shm DeviceMajor:0 DeviceMinor:262 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:296 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d9e7ae5f40305fe55fb0cabd24c3de09ea3562e9ade8d9cbbe11959a2e87f0/userdata/shm DeviceMajor:0 DeviceMinor:353 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:246 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~projected/kube-api-access-sg62n DeviceMajor:0 DeviceMinor:343 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:399 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:401 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-491 DeviceMajor:0 DeviceMinor:491 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~projected/kube-api-access-rhbcn DeviceMajor:0 DeviceMinor:252 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:391 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-433 DeviceMajor:0 DeviceMinor:433 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b36b9d5-51b0-4038-ae52-a7d1299cd284/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/549ff32d4bfc5fdfdc8cf55ca26367dc4a0d7142c3b0eb9b4f50446251e11088/userdata/shm DeviceMajor:0 DeviceMinor:266 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:423 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-431 DeviceMajor:0 DeviceMinor:431 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-448 DeviceMajor:0 DeviceMinor:448 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-511 DeviceMajor:0 DeviceMinor:511 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6e729acdcd98cfbd690797587f53bbe344eacf4d7008bac5b2745e7dc110d7/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:400 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2/userdata/shm DeviceMajor:0 DeviceMinor:406 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:342 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:351 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/da158b42-504b-4dfa-9b41-3333f4bbf0e7/volumes/kubernetes.io~projected/kube-api-access-n9xx9 DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50/volumes/kubernetes.io~projected/kube-api-access-5hp9b DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:253 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:255 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~projected/kube-api-access-6pj85 DeviceMajor:0 DeviceMinor:298 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:397 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:261 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a72a542-4dd5-46f1-bdba-e82aba26d948/volumes/kubernetes.io~projected/kube-api-access-87sz4 DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8d4333e0-6941-4815-844e-b7641b1ae720/volumes/kubernetes.io~projected/kube-api-access-qrksh DeviceMajor:0 DeviceMinor:60 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:396 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a396a10-b996-4d83-9783-f4a6a543ff6e/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:251 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:292 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/525049c6-49f7-46f1-8f0e-b31091e03f26/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:336 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8690d85665268fdd8145b3348f50a61b8d06d9f945f0ee2d77e89eaa9b1fe333/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:256 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:294 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83/userdata/shm DeviceMajor:0 DeviceMinor:429 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d84833dcdc34f6f28c2b089615b8e89cfc6743c77375fdfeaf4fd669e5b9b693/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b36b9d5-51b0-4038-ae52-a7d1299cd284/volumes/kubernetes.io~projected/kube-api-access-7mgmz DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24ca9cdd0a7ca2af7911d75288641eec6fb6dec9ceb9ac58858a6dd87bcf1310/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:290 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:291 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:397 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~projected/kube-api-access-df7bw DeviceMajor:0 DeviceMinor:265 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:349 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:440 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/8d4333e0-6941-4815-844e-b7641b1ae720/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:464 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f687b68853e2e396f8c6fcef9b6dad5ab49df7d6fdc025dcbd9f653316ed785/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/acb961211694eb105ce98bef0db1ee2ece382a20c7953d03b2cd72a82f901ef3/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e5f527d-4282-4ace-b004-5a2504a1d73e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa997701-363d-4c89-98f7-b6e5c349b892/volumes/kubernetes.io~projected/kube-api-access-48bzs DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~projected/kube-api-access-h2k5h DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/volumes/kubernetes.io~projected/kube-api-access-prsvj DeviceMajor:0 DeviceMinor:258 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d201227-679c-4342-8bbc-f386d540cc81/volumes/kubernetes.io~projected/kube-api-access-mslsf DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:290 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~projected/kube-api-access-4kpnl DeviceMajor:0 DeviceMinor:405 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9292ef355064861b1536b9b3d4bb5d0bee017c91ef8d00cae318d56817eb311/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/volumes/kubernetes.io~projected/kube-api-access-cp8wt DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:297 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-373 DeviceMajor:0 DeviceMinor:373 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df470bfade8f36219972b1294f0f861dd870600dcb85ca496681a71ce5f7a33f/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:263 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-486 DeviceMajor:0 DeviceMinor:486 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdb37df7daee8202378273f067aeb29605d4a2bc251661526b5d365b58646ba0/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4f8be0f7-d4c4-4bc9-ae48-ef845d362377/volumes/kubernetes.io~projected/kube-api-access-7nzhv DeviceMajor:0 DeviceMinor:465 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:254 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:435 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2e9a5db-d5d7-4b26-828a-203c6557820e/volumes/kubernetes.io~projected/kube-api-access-bcfzz DeviceMajor:0 DeviceMinor:61 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:397 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:402 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:397 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c36de38-25e2-4eb1-9b01-a7c342084792/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:460 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:295 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:348 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e522e488059a8c2d0ba51f94605e25406cff59318ee036f1e13e3c9414b11de6/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-418 DeviceMajor:0 DeviceMinor:418 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f178ba1f-4d58-4164-862d-0b99e5ffa859/volumes/kubernetes.io~projected/kube-api-access-4xcnn DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/985a4a048c55c5876c5f1ce833ec4eed17203648d78b0d4f73c7f65d0a789d25/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c73bcc3e26c17d123280dceb14916054cdd09f5f3af9f2bd0d2e9e9d0b60c5e6/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5e0152b3-9c26-4191-bfaf-182715417b9b/volumes/kubernetes.io~projected/kube-api-access-48hlf DeviceMajor:0 DeviceMinor:352 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:398 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:427 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-496 DeviceMajor:0 DeviceMinor:496 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:09588104d7b37b9 MacAddress:16:a6:ad:ea:c0:de Speed:10000 Mtu:8901} {Name:162b4a4f77cfd1e MacAddress:2e:5b:39:87:92:ec Speed:10000 Mtu:8901} {Name:24ca9cdd0a7ca2a MacAddress:16:e7:bd:ef:7b:84 Speed:10000 Mtu:8901} {Name:52d9e7ae5f40305 MacAddress:d6:9e:0b:15:16:e4 Speed:10000 Mtu:8901} {Name:5b28e778765e389 MacAddress:2a:6d:f2:b9:11:b8 Speed:10000 Mtu:8901} {Name:5f687b68853e2e3 MacAddress:ae:26:f0:2c:0e:03 Speed:10000 Mtu:8901} {Name:8690d85665268fd MacAddress:b2:19:1f:9d:0e:07 Speed:10000 Mtu:8901} {Name:a496ebc2ccba318 MacAddress:e6:98:92:b6:63:95 Speed:10000 Mtu:8901} {Name:b4feb442d6e2eb9 MacAddress:66:30:26:70:8c:9f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:4a:1c:d7 Speed:0 Mtu:9001} {Name:br-int MacAddress:d6:9a:94:40:72:35 Speed:0 Mtu:8901} {Name:cc606ad47b1e3eb MacAddress:f6:e2:ba:df:dd:10 Speed:10000 Mtu:8901} {Name:df470bfade8f362 MacAddress:0a:82:bd:89:31:d8 Speed:10000 Mtu:8901} {Name:df6ee019439ff0e MacAddress:a6:07:56:c2:98:cd Speed:10000 Mtu:8901} {Name:e978785d0606888 MacAddress:26:08:06:b9:ab:f6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:4a:1c:d7 Speed:0 Mtu:9001} {Name:fbc64dc6a549cbc MacAddress:b2:70:1f:30:dc:80 Speed:10000 Mtu:8901} {Name:fdb37df7daee820 MacAddress:ae:63:8c:c4:16:a8 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fa:58:b7:12:fd:7d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:22:74:a6:70:97:76 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 14:48:16.099515 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099123 6554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 14:48:16.099515 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099165 6554 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 14:48:16.099624 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099514 6554 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 14:48:16.099717 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099550 6554 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-204.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 14:48:16.099794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099720 6554 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 14:48:16.099794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099731 6554 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 14:48:16.099794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099745 6554 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:48:16.099794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099770 6554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 14:48:16.099961 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.099858 6554 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 14:48:16.100069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100056 6554 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 14:48:16.100194 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100183 6554 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 14:48:16.100249 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100227 6554 kubelet.go:493] "Attempting to sync node with API server" Oct 30 14:48:16.100249 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100247 6554 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 14:48:16.100334 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100262 6554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 14:48:16.100334 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100275 6554 kubelet.go:405] "Adding apiserver pod source" Oct 30 14:48:16.100334 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.100286 6554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 14:48:16.101019 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.101002 6554 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 14:48:16.101019 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.101017 6554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:48:16.101523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.101501 6554 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 14:48:16.104532 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104517 6554 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 14:48:16.104532 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104530 6554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 14:48:16.104634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104625 6554 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 14:48:16.104810 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104798 6554 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 14:48:16.104841 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104828 6554 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 14:48:16.104932 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104923 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 14:48:16.104963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104939 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 14:48:16.104963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104945 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 14:48:16.104963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104950 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 14:48:16.104963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104958 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104966 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104976 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104982 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104988 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.104994 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.105002 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.105012 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.105036 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 14:48:16.105069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.105042 6554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 14:48:16.108287 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.108272 6554 server.go:1263] "Started kubelet" Oct 30 14:48:16.109024 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.108366 6554 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 14:48:16.109024 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.108382 6554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 14:48:16.109024 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.108453 6554 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 14:48:16.109024 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.108805 6554 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 14:48:16.108819 ip-10-0-139-204 systemd[1]: Started Kubernetes Kubelet. Oct 30 14:48:16.113413 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.113393 6554 server.go:317] "Adding debug handlers to kubelet server" Oct 30 14:48:16.115302 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.115282 6554 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 14:48:16.116128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.115993 6554 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 14:48:16.116128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.116003 6554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 14:48:16.116128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.116025 6554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 14:48:16.116128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.116013 6554 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 14:48:16.116128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.116091 6554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 14:42:06 +0000 UTC" deadline="2028-05-08 18:45:07.121544663 +0000 UTC" Oct 30 14:48:16.116128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.116112 6554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13347h56m51.005438093s" Oct 30 14:48:16.116648 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.116419 6554 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-204.ec2.internal\" not found" Oct 30 14:48:16.117337 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117320 6554 factory.go:153] Registering CRI-O factory Oct 30 14:48:16.117441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117340 6554 factory.go:223] Registration of the crio container factory successfully Oct 30 14:48:16.117441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117419 6554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 14:48:16.117441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117430 6554 factory.go:55] Registering systemd factory Oct 30 14:48:16.117441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117438 6554 factory.go:223] Registration of the systemd container factory successfully Oct 30 14:48:16.117646 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117454 6554 factory.go:103] Registering Raw factory Oct 30 14:48:16.117646 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117466 6554 manager.go:1199] Started watching for new ooms in manager Oct 30 14:48:16.117830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.117817 6554 manager.go:319] Starting recovery of all containers Oct 30 14:48:16.118649 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118608 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a396a10-b996-4d83-9783-f4a6a543ff6e" volumeName="kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118651 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a72a542-4dd5-46f1-bdba-e82aba26d948" volumeName="kubernetes.io/projected/5a72a542-4dd5-46f1-bdba-e82aba26d948-kube-api-access-87sz4" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118662 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118670 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4333e0-6941-4815-844e-b7641b1ae720" volumeName="kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118678 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118690 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118701 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-federate-client-tls" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118726 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d4333e0-6941-4815-844e-b7641b1ae720" volumeName="kubernetes.io/projected/8d4333e0-6941-4815-844e-b7641b1ae720-kube-api-access-qrksh" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118736 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2e9a5db-d5d7-4b26-828a-203c6557820e" volumeName="kubernetes.io/projected/d2e9a5db-d5d7-4b26-828a-203c6557820e-kube-api-access-bcfzz" seLinuxMountContext="" Oct 30 14:48:16.118743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118743 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118751 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118759 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118766 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-client-tls" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118772 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118782 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118797 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118804 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d201227-679c-4342-8bbc-f386d540cc81" volumeName="kubernetes.io/configmap/6d201227-679c-4342-8bbc-f386d540cc81-iptables-alerter-script" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118812 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e5f527d-4282-4ace-b004-5a2504a1d73e" volumeName="kubernetes.io/secret/8e5f527d-4282-4ace-b004-5a2504a1d73e-agent-certs" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118819 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118825 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d" volumeName="kubernetes.io/projected/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-kube-api-access-h2k5h" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118832 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118839 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118846 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178ba1f-4d58-4164-862d-0b99e5ffa859" volumeName="kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-env-overrides" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118853 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118861 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee24becd-13c7-4fd5-a314-0b5f27a066ad" volumeName="kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cni-binary-copy" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118867 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118875 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118885 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118896 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e74bcbe-9984-4bc1-bff3-a19c3b0adb50" volumeName="kubernetes.io/projected/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-api-access-5hp9b" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118904 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/projected/5e0152b3-9c26-4191-bfaf-182715417b9b-kube-api-access-48hlf" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118912 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118930 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118938 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118945 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118952 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcea1b1f-2eb6-466b-b72f-b1e7d6e50117" volumeName="kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118959 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118966 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13d07af5-d2a4-401e-9d97-9b98c4f6e2d8" volumeName="kubernetes.io/empty-dir/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-tmp-dir" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118974 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b36b9d5-51b0-4038-ae52-a7d1299cd284" volumeName="kubernetes.io/projected/2b36b9d5-51b0-4038-ae52-a7d1299cd284-kube-api-access-7mgmz" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118982 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118989 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/projected/525049c6-49f7-46f1-8f0e-b31091e03f26-kube-api-access-sg62n" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.118997 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcea1b1f-2eb6-466b-b72f-b1e7d6e50117" volumeName="kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119005 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119013 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c36de38-25e2-4eb1-9b01-a7c342084792" volumeName="kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119021 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b36b9d5-51b0-4038-ae52-a7d1299cd284" volumeName="kubernetes.io/secret/2b36b9d5-51b0-4038-ae52-a7d1299cd284-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119028 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119034 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119041 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa997701-363d-4c89-98f7-b6e5c349b892" volumeName="kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119049 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119056 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178ba1f-4d58-4164-862d-0b99e5ffa859" volumeName="kubernetes.io/projected/f178ba1f-4d58-4164-862d-0b99e5ffa859-kube-api-access-4xcnn" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119063 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e74bcbe-9984-4bc1-bff3-a19c3b0adb50" volumeName="kubernetes.io/empty-dir/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-data-volume" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119072 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119080 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a72a542-4dd5-46f1-bdba-e82aba26d948" volumeName="kubernetes.io/configmap/5a72a542-4dd5-46f1-bdba-e82aba26d948-serviceca" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119087 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119095 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119103 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" volumeName="kubernetes.io/projected/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-kube-api-access-b9lcg" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119110 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13d07af5-d2a4-401e-9d97-9b98c4f6e2d8" volumeName="kubernetes.io/configmap/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-config-volume" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119122 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e74bcbe-9984-4bc1-bff3-a19c3b0adb50" volumeName="kubernetes.io/secret/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119128 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119139 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" volumeName="kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119147 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178ba1f-4d58-4164-862d-0b99e5ffa859" volumeName="kubernetes.io/secret/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119155 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d" volumeName="kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-tuned" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119163 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119173 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119180 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119187 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119197 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e74bcbe-9984-4bc1-bff3-a19c3b0adb50" volumeName="kubernetes.io/configmap/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119211 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119222 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcea1b1f-2eb6-466b-b72f-b1e7d6e50117" volumeName="kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:16.119189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119232 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d201227-679c-4342-8bbc-f386d540cc81" volumeName="kubernetes.io/projected/6d201227-679c-4342-8bbc-f386d540cc81-kube-api-access-mslsf" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119242 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119254 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" volumeName="kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119268 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119280 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119294 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da158b42-504b-4dfa-9b41-3333f4bbf0e7" volumeName="kubernetes.io/empty-dir/da158b42-504b-4dfa-9b41-3333f4bbf0e7-tmp-dir" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119306 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119317 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178ba1f-4d58-4164-862d-0b99e5ffa859" volumeName="kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-script-lib" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119328 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119340 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa997701-363d-4c89-98f7-b6e5c349b892" volumeName="kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119371 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a396a10-b996-4d83-9783-f4a6a543ff6e" volumeName="kubernetes.io/configmap/0a396a10-b996-4d83-9783-f4a6a543ff6e-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119383 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119394 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119404 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/empty-dir/525049c6-49f7-46f1-8f0e-b31091e03f26-audit-log" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119417 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119428 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119441 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119468 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119480 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="525049c6-49f7-46f1-8f0e-b31091e03f26" volumeName="kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-client-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119490 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119502 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" volumeName="kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119513 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d" volumeName="kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-tmp" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119526 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119536 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119547 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119557 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e5f527d-4282-4ace-b004-5a2504a1d73e" volumeName="kubernetes.io/configmap/8e5f527d-4282-4ace-b004-5a2504a1d73e-konnectivity-ca" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119569 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119583 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa997701-363d-4c89-98f7-b6e5c349b892" volumeName="kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-binary-copy" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119596 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa997701-363d-4c89-98f7-b6e5c349b892" volumeName="kubernetes.io/projected/aa997701-363d-4c89-98f7-b6e5c349b892-kube-api-access-48bzs" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119610 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119621 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119633 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f178ba1f-4d58-4164-862d-0b99e5ffa859" volumeName="kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119646 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a396a10-b996-4d83-9783-f4a6a543ff6e" volumeName="kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119661 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119675 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119687 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119699 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e0152b3-9c26-4191-bfaf-182715417b9b" volumeName="kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119710 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88a21bd0-b038-400e-8e85-be20c12f8ec0" volumeName="kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119723 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da158b42-504b-4dfa-9b41-3333f4bbf0e7" volumeName="kubernetes.io/projected/da158b42-504b-4dfa-9b41-3333f4bbf0e7-kube-api-access-n9xx9" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119736 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a396a10-b996-4d83-9783-f4a6a543ff6e" volumeName="kubernetes.io/projected/0a396a10-b996-4d83-9783-f4a6a543ff6e-kube-api-access-rhbcn" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119748 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" volumeName="kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119760 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119774 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119786 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" volumeName="kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-metrics-client-ca" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119819 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13d07af5-d2a4-401e-9d97-9b98c4f6e2d8" volumeName="kubernetes.io/projected/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-kube-api-access-cp8wt" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119841 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119855 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" volumeName="kubernetes.io/empty-dir/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-textfile" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119867 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee24becd-13c7-4fd5-a314-0b5f27a066ad" volumeName="kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-daemon-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119881 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13d07af5-d2a4-401e-9d97-9b98c4f6e2d8" volumeName="kubernetes.io/secret/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-metrics-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119894 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b69c00-6f7c-4048-aa22-c388091fdec2" volumeName="kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119905 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcea1b1f-2eb6-466b-b72f-b1e7d6e50117" volumeName="kubernetes.io/projected/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-api-access-prsvj" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119917 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcea1b1f-2eb6-466b-b72f-b1e7d6e50117" volumeName="kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119930 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119940 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcea1b1f-2eb6-466b-b72f-b1e7d6e50117" volumeName="kubernetes.io/empty-dir/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-volume-directive-shadow" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119953 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e36a4793-6b81-439f-b357-cc7e80803eb0" volumeName="kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119965 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ee24becd-13c7-4fd5-a314-0b5f27a066ad" volumeName="kubernetes.io/projected/ee24becd-13c7-4fd5-a314-0b5f27a066ad-kube-api-access-mfq98" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119976 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="393a4718-b786-4ac2-b4b1-cdb250c57261" volumeName="kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.119988 6554 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f8be0f7-d4c4-4bc9-ae48-ef845d362377" volumeName="kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv" seLinuxMountContext="" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.120001 6554 reconstruct.go:97] "Volume reconstruction finished" Oct 30 14:48:16.121523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.120008 6554 reconciler.go:29] "Reconciler: start to sync state" Oct 30 14:48:16.144840 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.144811 6554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 14:48:16.146250 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.146232 6554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 14:48:16.146385 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.146253 6554 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 14:48:16.146385 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.146273 6554 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 14:48:16.146385 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.146311 6554 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 14:48:16.161017 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.160996 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="3136254ef0afc29d67ec4b720c557c528cd9985a0ba39473588d90fdfd96533d" exitCode=0 Oct 30 14:48:16.163147 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.163123 6554 generic.go:356] "Generic (PLEG): container finished" podID="e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076" containerID="0b5b55dce5a9daeaf938649a055d740b89766384968afa7889fc1a961b574ff6" exitCode=0 Oct 30 14:48:16.170556 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.170536 6554 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:16.170641 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.170610 6554 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 14:48:16.170688 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.170677 6554 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:16.186621 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.186599 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" exitCode=0 Oct 30 14:48:16.196812 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.196695 6554 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:16.197173 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.197114 6554 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 14:48:16.201977 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.201943 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="760bbb81d9d24c37246230c8d22687c16f60f118ecdfa5039f46c087ef602fce" exitCode=0 Oct 30 14:48:16.201977 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.201964 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="faa028b33643f77b6a9776aedf98dda1605cb09ec3de2a098cc8c62e02693f67" exitCode=0 Oct 30 14:48:16.201977 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.201971 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="348eec614aecf18363d407d494b066af89d921e33d60c889107e7381c2e85487" exitCode=0 Oct 30 14:48:16.201977 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.201980 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="87e58bb1cf14d4d3c21e2d9f06c85438cc899b9c84fae5ae726774d96f1fb4d4" exitCode=0 Oct 30 14:48:16.202212 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.201993 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="ee2bebe6bcd0ced81a8d5a5c8cbc4aefee9ca1d6385d8ad6b3dceb714ecfbd1d" exitCode=0 Oct 30 14:48:16.202212 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.202002 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa997701-363d-4c89-98f7-b6e5c349b892" containerID="8460970d5660af7f41a8f6f4c79d342700c925263ed8b362f102160623f84d3a" exitCode=0 Oct 30 14:48:16.208817 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.208787 6554 generic.go:356] "Generic (PLEG): container finished" podID="4c8f5d16b96687b2a915ddaffb597231" containerID="ce4e069c6fe2a275f604bb87c661fd7f11a2af0000af8617b4e5c23d2dead5ff" exitCode=0 Oct 30 14:48:16.236983 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.236928 6554 manager.go:324] Recovery completed Oct 30 14:48:16.246396 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.246373 6554 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 14:48:16.261607 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261584 6554 cpu_manager.go:226] "Starting" policy="none" Oct 30 14:48:16.261607 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261600 6554 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 14:48:16.261741 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261618 6554 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 14:48:16.261741 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261725 6554 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 14:48:16.261836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261735 6554 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 14:48:16.261836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261752 6554 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 14:48:16.261836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261757 6554 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 14:48:16.261836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261762 6554 policy_none.go:50] "Start" Oct 30 14:48:16.261836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261771 6554 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 14:48:16.261836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.261780 6554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 14:48:16.262910 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.262898 6554 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 14:48:16.262910 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.262910 6554 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 14:48:16.262972 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.262916 6554 policy_none.go:44] "Start" Oct 30 14:48:16.267773 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.267745 6554 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 14:48:16.270402 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.270389 6554 manager.go:341] "Starting Device Plugin manager" Oct 30 14:48:16.270857 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.270414 6554 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 14:48:16.270857 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.270858 6554 server.go:87] "Starting device plugin registration server" Oct 30 14:48:16.271136 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271123 6554 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 14:48:16.271169 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271138 6554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 14:48:16.271252 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271232 6554 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 14:48:16.271406 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271371 6554 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 14:48:16.271471 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271411 6554 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 14:48:16.271471 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271418 6554 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 14:48:16.271577 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.271480 6554 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T14:48:16.27140094Z","UUID":"249f9a09-1f1d-4e72-943e-9ffe9e0e7e97","Handler":null,"Name":"","Endpoint":""} Oct 30 14:48:16.272767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.272752 6554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 14:48:16.272853 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.272774 6554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 14:48:16.273282 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.273261 6554 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 14:48:16.371699 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.371683 6554 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 14:48:16.373512 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.373497 6554 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientMemory" Oct 30 14:48:16.373616 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.373524 6554 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 14:48:16.373616 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.373544 6554 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-204.ec2.internal" event="NodeHasSufficientPID" Oct 30 14:48:16.373616 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.373588 6554 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.412238 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.412219 6554 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.412321 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.412267 6554 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.446769 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.446725 6554 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal"] Oct 30 14:48:16.447332 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447302 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" event={"ID":"c47c2ff730e7510466e8747f4e5f45ca","Type":"ContainerStarted","Data":"9ea763045043bfb5c8b3e741b4b19ad9c84af466ffadba2738928187e0aa9cb5"} Oct 30 14:48:16.447435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447336 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" event={"ID":"c47c2ff730e7510466e8747f4e5f45ca","Type":"ContainerStarted","Data":"b08d07cc6e51163a956887df0232b47f64877a13f4faf55e3dd105d0392e457c"} Oct 30 14:48:16.447435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447335 6554 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.447435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447398 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"4c8f5d16b96687b2a915ddaffb597231","Type":"ContainerStarted","Data":"46c8575b097b7a12545892f8e3fc4e1084cadb08d422b2af0c400a66c9af8f5c"} Oct 30 14:48:16.447435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447408 6554 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.447435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447412 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"4c8f5d16b96687b2a915ddaffb597231","Type":"ContainerDied","Data":"ce4e069c6fe2a275f604bb87c661fd7f11a2af0000af8617b4e5c23d2dead5ff"} Oct 30 14:48:16.447435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.447425 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" event={"ID":"4c8f5d16b96687b2a915ddaffb597231","Type":"ContainerStarted","Data":"138ddb3ae19068462a8141f94f01a039c274dac8167a23d1532265c11b77cb66"} Oct 30 14:48:16.473746 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.473728 6554 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:16.473823 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.473761 6554 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.474346 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.474331 6554 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:16.474442 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:16.474392 6554 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.572701 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.572645 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.572701 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.572677 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.572803 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.572701 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c47c2ff730e7510466e8747f4e5f45ca-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"c47c2ff730e7510466e8747f4e5f45ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.673260 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.673236 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.673344 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.673265 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c47c2ff730e7510466e8747f4e5f45ca-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"c47c2ff730e7510466e8747f4e5f45ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.673344 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.673281 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.673344 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.673331 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.673468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.673336 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4c8f5d16b96687b2a915ddaffb597231-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" (UID: \"4c8f5d16b96687b2a915ddaffb597231\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:16.673468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:16.673372 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c47c2ff730e7510466e8747f4e5f45ca-config\") pod \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" (UID: \"c47c2ff730e7510466e8747f4e5f45ca\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:17.100912 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.100887 6554 apiserver.go:52] "Watching apiserver" Oct 30 14:48:17.164699 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.164661 6554 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 14:48:17.165633 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.165605 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-8829w","kube-system/global-pull-secret-syncer-bz4fj","openshift-multus/multus-hjchw","openshift-network-diagnostics/network-check-target-fnbqw","kube-system/konnectivity-agent-sbcnp","openshift-ingress-canary/ingress-canary-mnl72","openshift-multus/multus-additional-cni-plugins-rgsm9","openshift-ovn-kubernetes/ovnkube-node-wsrm2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal","openshift-monitoring/node-exporter-jkbnv","openshift-insights/insights-runtime-extractor-hj6kg","openshift-cluster-node-tuning-operator/tuned-dmr9v","openshift-console/console-b5c666f7f-p2trj","openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw","openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq","openshift-network-operator/iptables-alerter-hzqxz","openshift-monitoring/alertmanager-main-0","openshift-monitoring/prometheus-k8s-0","openshift-console/console-748747647-9rm5d","openshift-dns/dns-default-xmw9l","openshift-dns/node-resolver-q5b49","openshift-image-registry/node-ca-vvpcd","kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal","openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq","openshift-multus/network-metrics-daemon-cjwh8","openshift-console/console-56cf858bf9-pd4k9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2"] Oct 30 14:48:17.185166 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185110 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.185404 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185173 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 14:48:17.185404 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185228 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 14:48:17.185404 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185344 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 14:48:17.185682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185406 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.185682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185463 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.185682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185505 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.185682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185548 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 14:48:17.185682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185654 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185723 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185756 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sf926\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185768 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185803 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185806 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185876 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 14:48:17.185921 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185889 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185935 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185973 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8r54d\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.185984 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-b74bf\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.186017 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.186057 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.186080 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g6822\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.186086 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fb2kz\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.186091 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-p94r5\"" Oct 30 14:48:17.186258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.186178 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2zmnb\"" Oct 30 14:48:17.189122 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189099 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.189281 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189230 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 14:48:17.189567 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189545 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.189686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189655 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 14:48:17.189686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189578 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bksxs\"" Oct 30 14:48:17.189686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189634 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 14:48:17.189929 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189596 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-4bwjl\"" Oct 30 14:48:17.189929 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189731 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 14:48:17.189929 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.189792 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-52rrx\"" Oct 30 14:48:17.190974 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.190922 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 14:48:17.191116 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191096 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.191225 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191100 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 14:48:17.191225 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191150 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.191225 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191167 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.191392 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191373 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.191524 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191395 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 14:48:17.191524 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191426 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.191752 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191724 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.191828 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191775 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 14:48:17.191828 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191778 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.191828 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191789 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.191828 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.191792 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.192040 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.192022 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 14:48:17.192578 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.192560 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.192645 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.192593 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.192692 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.192647 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 14:48:17.193120 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.193108 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-26d46\"" Oct 30 14:48:17.193774 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.193757 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 14:48:17.193854 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.193774 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 14:48:17.194086 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.194066 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-b4vks\"" Oct 30 14:48:17.194161 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.194086 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cd47w\"" Oct 30 14:48:17.194161 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.194098 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 14:48:17.194161 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.194123 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-jc64h\"" Oct 30 14:48:17.194161 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.194100 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nzpht\"" Oct 30 14:48:17.195888 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.195872 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 14:48:17.196056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.196023 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rw2pz\"" Oct 30 14:48:17.196108 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.196055 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.196108 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.196067 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 14:48:17.196163 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.196117 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ss7nc\"" Oct 30 14:48:17.198788 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.198770 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 14:48:17.214034 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.214010 6554 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:17.214158 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.214143 6554 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:17.219129 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.219114 6554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 14:48:17.243334 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.243312 6554 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:17.243455 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.243365 6554 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 14:48:17.243455 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:17.243374 6554 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-204.ec2.internal" Oct 30 14:48:17.243455 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:17.243401 6554 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-204.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-204.ec2.internal" Oct 30 14:48:17.275951 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.275932 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.275951 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.275957 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b36b9d5-51b0-4038-ae52-a7d1299cd284-cert\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:48:17.276089 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.275980 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.276089 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.275999 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-run\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.276089 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276018 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.276251 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276131 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.276251 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276208 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.276251 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276232 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrksh\" (UniqueName: \"kubernetes.io/projected/8d4333e0-6941-4815-844e-b7641b1ae720-kube-api-access-qrksh\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:17.276251 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276246 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2b36b9d5-51b0-4038-ae52-a7d1299cd284-cert\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:48:17.276475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276256 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-socket-dir-parent\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.276475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276286 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-var-lib-kubelet\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.276475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276344 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.276475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276402 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.276475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276432 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.276475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276463 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.276686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276487 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.276686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276506 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.276686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276515 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-var-lib-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.276686 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276563 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-bin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.276830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276732 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.276830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276748 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-accelerators-collector-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.276830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276769 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.276830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276782 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.276830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276800 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-federate-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276834 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276860 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d201227-679c-4342-8bbc-f386d540cc81-host-slash\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276885 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276904 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcfzz\" (UniqueName: \"kubernetes.io/projected/d2e9a5db-d5d7-4b26-828a-203c6557820e-kube-api-access-bcfzz\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276920 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.276948 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.277033 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277000 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.277291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277125 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-slash\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.277291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277156 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.277291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277170 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-node-log\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.277291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277191 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.277291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277230 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-env-overrides\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.277291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277267 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.277497 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277305 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-k8s-cni-cncf-io\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.277497 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277379 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-etc-kubernetes\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.277497 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277413 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.277497 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277447 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.277497 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277452 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-env-overrides\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277550 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277571 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277634 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/525049c6-49f7-46f1-8f0e-b31091e03f26-audit-log\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277663 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277691 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277717 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-multus-certs\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.277736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277730 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/525049c6-49f7-46f1-8f0e-b31091e03f26-audit-log\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277743 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277778 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-tmp\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277829 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-tmp\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277838 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a72a542-4dd5-46f1-bdba-e82aba26d948-host\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277868 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-metrics-server-audit-profiles\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277907 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-serving-certs-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277934 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277958 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-config\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.277985 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-modprobe-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278013 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-systemd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.278056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278039 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278092 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278123 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-tuned\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278147 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-config\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278213 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-tuned\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278242 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278286 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da158b42-504b-4dfa-9b41-3333f4bbf0e7-hosts-file\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278300 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278324 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9lcg\" (UniqueName: \"kubernetes.io/projected/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-kube-api-access-b9lcg\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278345 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278380 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-kubelet\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278396 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e5f527d-4282-4ace-b004-5a2504a1d73e-agent-certs\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278413 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-metrics-tls\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.278450 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278432 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278465 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278493 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278520 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-metrics-client-ca\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278517 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278546 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278573 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbcn\" (UniqueName: \"kubernetes.io/projected/0a396a10-b996-4d83-9783-f4a6a543ff6e-kube-api-access-rhbcn\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278585 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278610 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278651 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-metrics-tls\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278653 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278695 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e5f527d-4282-4ace-b004-5a2504a1d73e-agent-certs\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278704 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278729 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-config-volume\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278754 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-tmp-dir\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278760 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-metrics-client-ca\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278781 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278805 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278846 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-netns\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278855 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-tmp-dir\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278875 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278893 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278921 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.278923 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278929 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-config-volume\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.278948 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-client-certs\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279040 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279075 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-etc-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279100 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-netd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279125 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279187 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-root\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279219 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279242 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgmz\" (UniqueName: \"kubernetes.io/projected/2b36b9d5-51b0-4038-ae52-a7d1299cd284-kube-api-access-7mgmz\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279318 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279328 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279374 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-bin\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279408 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovn-node-metrics-cert\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279437 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prsvj\" (UniqueName: \"kubernetes.io/projected/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-api-access-prsvj\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279462 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279489 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279514 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-netns\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279541 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279565 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279603 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279609 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovn-node-metrics-cert\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279633 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279662 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-system-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279688 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cni-binary-copy\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279722 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279784 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hp9b\" (UniqueName: \"kubernetes.io/projected/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-api-access-5hp9b\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279813 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.279825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279830 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279846 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-multus\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279868 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a72a542-4dd5-46f1-bdba-e82aba26d948-serviceca\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279886 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1c36de38-25e2-4eb1-9b01-a7c342084792-original-pull-secret\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279902 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-client-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279915 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cni-binary-copy\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279926 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.279957 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280001 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280031 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280051 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280066 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280081 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-system-cni-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280107 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280148 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280152 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280149 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280184 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280211 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280230 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280246 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f527d-4282-4ace-b004-5a2504a1d73e-konnectivity-ca\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280267 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-sys\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280294 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280321 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-metrics-client-ca\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280369 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280398 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280424 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-ovn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280452 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysconfig\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280477 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-conf\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280509 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2k5h\" (UniqueName: \"kubernetes.io/projected/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-kube-api-access-h2k5h\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280526 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e5f527d-4282-4ace-b004-5a2504a1d73e-konnectivity-ca\") pod \"konnectivity-agent-sbcnp\" (UID: \"8e5f527d-4282-4ace-b004-5a2504a1d73e\") " pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280537 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280576 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-metrics-client-ca\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280586 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a72a542-4dd5-46f1-bdba-e82aba26d948-serviceca\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280623 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280656 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9xx9\" (UniqueName: \"kubernetes.io/projected/da158b42-504b-4dfa-9b41-3333f4bbf0e7-kube-api-access-n9xx9\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.280673 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280686 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-dbus\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280714 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48bzs\" (UniqueName: \"kubernetes.io/projected/aa997701-363d-4c89-98f7-b6e5c349b892-kube-api-access-48bzs\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280750 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280762 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280797 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-tls\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280826 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hlf\" (UniqueName: \"kubernetes.io/projected/5e0152b3-9c26-4191-bfaf-182715417b9b-kube-api-access-48hlf\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280855 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280915 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280953 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.280952 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-kubelet\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281003 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-script-lib\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281027 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-cnibin\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281054 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281093 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-device-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281113 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da158b42-504b-4dfa-9b41-3333f4bbf0e7-tmp-dir\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281129 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-wtmp\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281151 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d4333e0-6941-4815-844e-b7641b1ae720-metrics-certs\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281167 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f178ba1f-4d58-4164-862d-0b99e5ffa859-ovnkube-script-lib\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281155 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281205 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-systemd-units\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281215 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aa997701-363d-4c89-98f7-b6e5c349b892-cni-binary-copy\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281222 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da158b42-504b-4dfa-9b41-3333f4bbf0e7-tmp-dir\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281274 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-log-socket\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281292 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-lib-modules\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281312 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281366 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281395 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281413 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281431 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cnibin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281445 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-hostroot\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281465 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-daemon-config\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281491 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-systemd\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281492 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281559 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281601 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281613 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-daemon-config\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281638 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-textfile\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281660 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-kubelet-config\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281705 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281721 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-textfile\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281784 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281801 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281822 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281850 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-conf-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281877 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281905 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-tls\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281956 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d201227-679c-4342-8bbc-f386d540cc81-iptables-alerter-script\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.281990 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a396a10-b996-4d83-9783-f4a6a543ff6e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282018 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-kubernetes\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282045 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282074 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp8wt\" (UniqueName: \"kubernetes.io/projected/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-kube-api-access-cp8wt\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282101 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282144 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d201227-679c-4342-8bbc-f386d540cc81-iptables-alerter-script\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282148 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87sz4\" (UniqueName: \"kubernetes.io/projected/5a72a542-4dd5-46f1-bdba-e82aba26d948-kube-api-access-87sz4\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282151 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a396a10-b996-4d83-9783-f4a6a543ff6e-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282176 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-data-volume\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282220 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-sys\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282226 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-data-volume\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282266 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282303 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg62n\" (UniqueName: \"kubernetes.io/projected/525049c6-49f7-46f1-8f0e-b31091e03f26-kube-api-access-sg62n\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282337 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mslsf\" (UniqueName: \"kubernetes.io/projected/6d201227-679c-4342-8bbc-f386d540cc81-kube-api-access-mslsf\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282428 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.282433 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282457 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282486 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfq98\" (UniqueName: \"kubernetes.io/projected/ee24becd-13c7-4fd5-a314-0b5f27a066ad-kube-api-access-mfq98\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282513 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-host\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282542 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282569 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-crio-socket\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282598 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xcnn\" (UniqueName: \"kubernetes.io/projected/f178ba1f-4d58-4164-862d-0b99e5ffa859-kube-api-access-4xcnn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282627 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282636 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.284081 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.282645 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.285839 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.285815 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 14:48:17.285918 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.285872 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 14:48:17.286045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286027 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 14:48:17.286045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286040 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 14:48:17.286130 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286085 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 14:48:17.286173 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286162 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 14:48:17.286217 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286201 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 14:48:17.286320 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286305 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286326 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0a396a10-b996-4d83-9783-f4a6a543ff6e-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286818 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286820 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.286869 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.287037 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.287096 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.287251 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.287487 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.287879 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9w6js\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288066 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288302 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-metrics-server-audit-profiles\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288532 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lj7tc\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288545 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7mvkp\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288573 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 14:48:17.288949 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288852 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 14:48:17.289647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.288994 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 14:48:17.289647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289048 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 14:48:17.289647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289116 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 14:48:17.289647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289238 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-client-certs\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.289647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289412 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 14:48:17.289647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289449 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ctc47fv49jduf\"" Oct 30 14:48:17.289912 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289661 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.289912 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289689 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 14:48:17.289912 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289883 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 14:48:17.290047 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.289952 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290413 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290549 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290578 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ng4tk\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290652 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290750 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290747 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.290844 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ndmfj\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291015 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291091 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tcq4s\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291246 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291458 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291638 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291701 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291752 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-881julv4d1pcu\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291869 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291915 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292033 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.291773 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292157 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-tls\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292175 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292217 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292228 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.292327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292242 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292380 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292405 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292493 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292597 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292660 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292794 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292828 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292848 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292961 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-client-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.292975 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293021 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293031 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293051 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5e0152b3-9c26-4191-bfaf-182715417b9b-federate-client-tls\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293068 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293142 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293209 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293239 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293248 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/525049c6-49f7-46f1-8f0e-b31091e03f26-secret-metrics-server-tls\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293323 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.293409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293361 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.294016 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293484 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.294016 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293542 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.294016 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293556 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.294016 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293567 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.294016 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293582 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.294016 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.293668 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.296515 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.296497 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.296836 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.296819 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.296958 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.296945 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.298685 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.298671 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.298751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.298732 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-serving-certs-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.298815 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.298798 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.298899 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.298886 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.298965 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.298950 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.299001 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.298981 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.299144 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.299127 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.300117 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.300102 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.300229 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.300218 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.300454 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.300440 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/525049c6-49f7-46f1-8f0e-b31091e03f26-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.300609 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.300586 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.302482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.301618 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 14:48:17.302482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.301636 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 14:48:17.302482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.301655 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 14:48:17.302482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302261 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrksh\" (UniqueName: \"kubernetes.io/projected/8d4333e0-6941-4815-844e-b7641b1ae720-kube-api-access-qrksh\") pod \"network-metrics-daemon-cjwh8\" (UID: \"8d4333e0-6941-4815-844e-b7641b1ae720\") " pod="openshift-multus/network-metrics-daemon-cjwh8" Oct 30 14:48:17.302482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302312 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 14:48:17.302482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302455 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.302689 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302487 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.302689 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302517 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.302689 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302577 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcfzz\" (UniqueName: \"kubernetes.io/projected/d2e9a5db-d5d7-4b26-828a-203c6557820e-kube-api-access-bcfzz\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.302689 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302651 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.302800 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.302711 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 14:48:17.308687 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.308667 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.308933 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.308899 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.308933 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.308923 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e0152b3-9c26-4191-bfaf-182715417b9b-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.309444 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.309425 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") pod \"console-b5c666f7f-p2trj\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.309444 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.309434 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") pod \"console-56cf858bf9-pd4k9\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:17.311695 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.311673 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") pod \"console-748747647-9rm5d\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:17.311755 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.311690 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.312368 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.312334 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.314181 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.314160 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9lcg\" (UniqueName: \"kubernetes.io/projected/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-kube-api-access-b9lcg\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.314341 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.314326 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbcn\" (UniqueName: \"kubernetes.io/projected/0a396a10-b996-4d83-9783-f4a6a543ff6e-kube-api-access-rhbcn\") pod \"openshift-state-metrics-7bd7489957-8829w\" (UID: \"0a396a10-b996-4d83-9783-f4a6a543ff6e\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-8829w" Oct 30 14:48:17.326001 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.325971 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48bzs\" (UniqueName: \"kubernetes.io/projected/aa997701-363d-4c89-98f7-b6e5c349b892-kube-api-access-48bzs\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.326084 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.326022 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9xx9\" (UniqueName: \"kubernetes.io/projected/da158b42-504b-4dfa-9b41-3333f4bbf0e7-kube-api-access-n9xx9\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.327476 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.327455 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hp9b\" (UniqueName: \"kubernetes.io/projected/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-kube-api-access-5hp9b\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.327595 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.327495 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfq98\" (UniqueName: \"kubernetes.io/projected/ee24becd-13c7-4fd5-a314-0b5f27a066ad-kube-api-access-mfq98\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.328298 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.328280 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2k5h\" (UniqueName: \"kubernetes.io/projected/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-kube-api-access-h2k5h\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.328701 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.328686 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgmz\" (UniqueName: \"kubernetes.io/projected/2b36b9d5-51b0-4038-ae52-a7d1299cd284-kube-api-access-7mgmz\") pod \"ingress-canary-mnl72\" (UID: \"2b36b9d5-51b0-4038-ae52-a7d1299cd284\") " pod="openshift-ingress-canary/ingress-canary-mnl72" Oct 30 14:48:17.331197 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.331180 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzhv\" (UniqueName: \"kubernetes.io/projected/4f8be0f7-d4c4-4bc9-ae48-ef845d362377-kube-api-access-7nzhv\") pod \"network-check-target-fnbqw\" (UID: \"4f8be0f7-d4c4-4bc9-ae48-ef845d362377\") " pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:17.331595 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.331581 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp8wt\" (UniqueName: \"kubernetes.io/projected/13d07af5-d2a4-401e-9d97-9b98c4f6e2d8-kube-api-access-cp8wt\") pod \"dns-default-xmw9l\" (UID: \"13d07af5-d2a4-401e-9d97-9b98c4f6e2d8\") " pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.333770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.333731 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hlf\" (UniqueName: \"kubernetes.io/projected/5e0152b3-9c26-4191-bfaf-182715417b9b-kube-api-access-48hlf\") pod \"telemeter-client-5d6ccfcd7f-rs5lq\" (UID: \"5e0152b3-9c26-4191-bfaf-182715417b9b\") " pod="openshift-monitoring/telemeter-client-5d6ccfcd7f-rs5lq" Oct 30 14:48:17.334044 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.334008 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg62n\" (UniqueName: \"kubernetes.io/projected/525049c6-49f7-46f1-8f0e-b31091e03f26-kube-api-access-sg62n\") pod \"metrics-server-6cbbd6bfb5-h9xsq\" (UID: \"525049c6-49f7-46f1-8f0e-b31091e03f26\") " pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:17.334206 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.334186 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xcnn\" (UniqueName: \"kubernetes.io/projected/f178ba1f-4d58-4164-862d-0b99e5ffa859-kube-api-access-4xcnn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.334396 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.334380 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prsvj\" (UniqueName: \"kubernetes.io/projected/dcea1b1f-2eb6-466b-b72f-b1e7d6e50117-kube-api-access-prsvj\") pod \"kube-state-metrics-5dc8d7d89b-9r8cw\" (UID: \"dcea1b1f-2eb6-466b-b72f-b1e7d6e50117\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-9r8cw" Oct 30 14:48:17.334466 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.334424 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mslsf\" (UniqueName: \"kubernetes.io/projected/6d201227-679c-4342-8bbc-f386d540cc81-kube-api-access-mslsf\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.335761 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.335743 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") pod \"alertmanager-main-0\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:17.337283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.337263 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87sz4\" (UniqueName: \"kubernetes.io/projected/5a72a542-4dd5-46f1-bdba-e82aba26d948-kube-api-access-87sz4\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.383205 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383126 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-modprobe-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.383205 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383157 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-systemd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383205 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383173 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383205 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383197 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da158b42-504b-4dfa-9b41-3333f4bbf0e7-hosts-file\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383220 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-kubelet\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383250 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-netns\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383268 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383273 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-etc-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383309 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-etc-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383309 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-modprobe-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383291 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da158b42-504b-4dfa-9b41-3333f4bbf0e7-hosts-file\") pod \"node-resolver-q5b49\" (UID: \"da158b42-504b-4dfa-9b41-3333f4bbf0e7\") " pod="openshift-dns/node-resolver-q5b49" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383325 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-netd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383373 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-root\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383383 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-kubelet\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383387 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-netd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383321 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-netns\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383248 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-systemd\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383400 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-bin\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383445 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-root\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383492 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-cni-bin\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383508 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-netns\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383534 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-system-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.383553 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383550 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383570 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-multus\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383602 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383610 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-system-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383627 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383631 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-netns\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383629 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-cni-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383654 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-system-cni-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383664 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383671 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-multus\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383684 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383687 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-system-cni-dir\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383691 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-sys\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383728 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-sys\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383770 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-ovn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383797 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysconfig\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383820 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-conf\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383835 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-run-ovn\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383847 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-dbus\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383866 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysconfig\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383874 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383901 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-kubelet\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383925 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-dbus\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383932 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-socket-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383926 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-cnibin\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383948 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-conf\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383957 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-device-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383962 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-kubelet\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383960 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aa997701-363d-4c89-98f7-b6e5c349b892-cnibin\") pod \"multus-additional-cni-plugins-rgsm9\" (UID: \"aa997701-363d-4c89-98f7-b6e5c349b892\") " pod="openshift-multus/multus-additional-cni-plugins-rgsm9" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383982 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-wtmp\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.383989 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-device-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384034 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-node-exporter-wtmp\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384061 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-systemd-units\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384081 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-log-socket\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384084 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-systemd-units\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384096 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-lib-modules\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384108 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-log-socket\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384113 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cnibin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384128 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-hostroot\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384146 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-cnibin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384142 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-systemd\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384174 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-hostroot\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384183 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384203 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-systemd\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384206 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-kubelet-config\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384214 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-lib-modules\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384229 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-conf-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384233 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384249 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-kubernetes\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384264 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384279 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-sys\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384295 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-kubernetes\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.384277 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384301 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-host\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384321 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076-sys\") pod \"node-exporter-jkbnv\" (UID: \"e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076\") " pod="openshift-monitoring/node-exporter-jkbnv" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384330 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-crio-socket\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384336 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-sys-fs\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384365 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-host\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384371 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-run\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384394 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-socket-dir-parent\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384400 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e74bcbe-9984-4bc1-bff3-a19c3b0adb50-crio-socket\") pod \"insights-runtime-extractor-hj6kg\" (UID: \"1e74bcbe-9984-4bc1-bff3-a19c3b0adb50\") " pod="openshift-insights/insights-runtime-extractor-hj6kg" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384399 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-conf-dir\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384407 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-var-lib-kubelet\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384424 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-var-lib-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384434 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-multus-socket-dir-parent\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384241 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1c36de38-25e2-4eb1-9b01-a7c342084792-kubelet-config\") pod \"global-pull-secret-syncer-bz4fj\" (UID: \"1c36de38-25e2-4eb1-9b01-a7c342084792\") " pod="kube-system/global-pull-secret-syncer-bz4fj" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384443 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-bin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384446 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-run\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384469 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384485 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-var-lib-cni-bin\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384488 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-var-lib-kubelet\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384494 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d201227-679c-4342-8bbc-f386d540cc81-host-slash\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384490 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-var-lib-openvswitch\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384504 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-etc-selinux\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384520 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-slash\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384526 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d201227-679c-4342-8bbc-f386d540cc81-host-slash\") pod \"iptables-alerter-hzqxz\" (UID: \"6d201227-679c-4342-8bbc-f386d540cc81\") " pod="openshift-network-operator/iptables-alerter-hzqxz" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384538 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-node-log\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384552 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384564 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-slash\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384571 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-k8s-cni-cncf-io\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384598 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-host-run-ovn-kubernetes\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384604 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f178ba1f-4d58-4164-862d-0b99e5ffa859-node-log\") pod \"ovnkube-node-wsrm2\" (UID: \"f178ba1f-4d58-4164-862d-0b99e5ffa859\") " pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384616 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-k8s-cni-cncf-io\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384622 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-etc-kubernetes\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384642 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384660 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-multus-certs\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384642 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-etc-kubernetes\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384672 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384684 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d2e9a5db-d5d7-4b26-828a-203c6557820e-registration-dir\") pod \"aws-ebs-csi-driver-node-j7wx2\" (UID: \"d2e9a5db-d5d7-4b26-828a-203c6557820e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j7wx2" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384704 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a72a542-4dd5-46f1-bdba-e82aba26d948-host\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384725 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ee24becd-13c7-4fd5-a314-0b5f27a066ad-host-run-multus-certs\") pod \"multus-hjchw\" (UID: \"ee24becd-13c7-4fd5-a314-0b5f27a066ad\") " pod="openshift-multus/multus-hjchw" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384755 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d-etc-sysctl-d\") pod \"tuned-dmr9v\" (UID: \"333a5e10-a1b0-48e3-a2d8-ff1c9221bb8d\") " pod="openshift-cluster-node-tuning-operator/tuned-dmr9v" Oct 30 14:48:17.385845 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.384776 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a72a542-4dd5-46f1-bdba-e82aba26d948-host\") pod \"node-ca-vvpcd\" (UID: \"5a72a542-4dd5-46f1-bdba-e82aba26d948\") " pod="openshift-image-registry/node-ca-vvpcd" Oct 30 14:48:17.494714 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.494686 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:17.513907 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.513882 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.518722 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.518700 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:17.597981 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.597957 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmw9l" Oct 30 14:48:17.919788 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.919748 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:17.939664 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:17.939630 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:20.093947 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:20.093906 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:20.565672 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:20.565642 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:48:23.711985 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:23.711947 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:23.712426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:23.712163 6554 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 14:48:29.387576 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:29.387502 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sbcnp" Oct 30 14:48:30.099864 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:30.099837 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:48:31.529982 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:31.529949 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cbbd6bfb5-h9xsq" Oct 30 14:48:31.816059 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:31.815989 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wsrm2" Oct 30 14:48:32.993896 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:32.993870 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dcc966694-t2bvg"] Oct 30 14:48:33.039205 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.039180 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcc966694-t2bvg"] Oct 30 14:48:33.039327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.039275 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.192820 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.192794 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-config\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.192938 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.192830 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-oauth-config\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.192938 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.192848 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-service-ca\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.193017 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.192986 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5hbg\" (UniqueName: \"kubernetes.io/projected/c6858670-0950-46e8-8db9-06e2f3b4aa24-kube-api-access-p5hbg\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.193017 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.193010 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-oauth-serving-cert\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.193108 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.193034 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-serving-cert\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.193161 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.193144 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6858670-0950-46e8-8db9-06e2f3b4aa24-tmp\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.193219 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.193193 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-trusted-ca-bundle\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294125 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294075 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-serving-cert\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294125 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294106 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6858670-0950-46e8-8db9-06e2f3b4aa24-tmp\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294287 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294127 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-trusted-ca-bundle\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294287 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294146 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-config\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294287 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294181 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-oauth-config\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294479 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294295 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-service-ca\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294479 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294394 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5hbg\" (UniqueName: \"kubernetes.io/projected/c6858670-0950-46e8-8db9-06e2f3b4aa24-kube-api-access-p5hbg\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294479 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294424 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-oauth-serving-cert\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294499 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c6858670-0950-46e8-8db9-06e2f3b4aa24-tmp\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.294682 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.294545 6554 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 14:48:33.295077 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.295050 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-config\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.295185 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.295172 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-service-ca\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.295268 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.295246 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-oauth-serving-cert\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.295428 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.295409 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6858670-0950-46e8-8db9-06e2f3b4aa24-trusted-ca-bundle\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.296800 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.296780 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-oauth-config\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.296986 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.296972 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c6858670-0950-46e8-8db9-06e2f3b4aa24-console-serving-cert\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.318931 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.318908 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5hbg\" (UniqueName: \"kubernetes.io/projected/c6858670-0950-46e8-8db9-06e2f3b4aa24-kube-api-access-p5hbg\") pod \"console-5dcc966694-t2bvg\" (UID: \"c6858670-0950-46e8-8db9-06e2f3b4aa24\") " pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.369751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.369728 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:33.696252 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:33.696228 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dcc966694-t2bvg"] Oct 30 14:48:33.698917 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:33.698889 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6858670_0950_46e8_8db9_06e2f3b4aa24.slice/crio-e368c3486614d9bba9233242734eb992102f8ce130056b74fffe72e4ecb101ab WatchSource:0}: Error finding container e368c3486614d9bba9233242734eb992102f8ce130056b74fffe72e4ecb101ab: Status 404 returned error can't find the container with id e368c3486614d9bba9233242734eb992102f8ce130056b74fffe72e4ecb101ab Oct 30 14:48:34.286144 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:34.286109 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc966694-t2bvg" event={"ID":"c6858670-0950-46e8-8db9-06e2f3b4aa24","Type":"ContainerStarted","Data":"a49d4cdf88658396fbdc636998f6a406417dd478e97e29a57d78735b30d5c599"} Oct 30 14:48:34.286592 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:34.286196 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dcc966694-t2bvg" event={"ID":"c6858670-0950-46e8-8db9-06e2f3b4aa24","Type":"ContainerStarted","Data":"e368c3486614d9bba9233242734eb992102f8ce130056b74fffe72e4ecb101ab"} Oct 30 14:48:34.333740 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:34.333695 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5dcc966694-t2bvg" podStartSLOduration=2.333682766 podStartE2EDuration="2.333682766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:34.332881615 +0000 UTC m=+18.291923019" watchObservedRunningTime="2026-10-30 14:48:34.333682766 +0000 UTC m=+18.292724162" Oct 30 14:48:37.247908 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.247867 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:37.248343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.248198 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="alertmanager" containerID="cri-o://b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" gracePeriod=120 Oct 30 14:48:37.248343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.248261 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-web" containerID="cri-o://99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" gracePeriod=120 Oct 30 14:48:37.248343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.248319 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy" containerID="cri-o://d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" gracePeriod=120 Oct 30 14:48:37.248343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.248319 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="prom-label-proxy" containerID="cri-o://b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" gracePeriod=120 Oct 30 14:48:37.248626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.248366 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="config-reloader" containerID="cri-o://980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" gracePeriod=120 Oct 30 14:48:37.248626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.248261 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-metric" containerID="cri-o://1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" gracePeriod=120 Oct 30 14:48:37.516482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.516419 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:37.627034 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627007 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627164 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627040 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627164 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627073 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627164 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627093 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627164 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627121 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627164 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627140 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627194 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627225 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627265 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627298 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627330 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627466 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.627771 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.627503 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") pod \"393a4718-b786-4ac2-b4b1-cdb250c57261\" (UID: \"393a4718-b786-4ac2-b4b1-cdb250c57261\") " Oct 30 14:48:37.628218 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.628184 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:37.629282 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.629249 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:37.629468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.629446 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:37.632043 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.632009 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.632713 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.632682 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "kube-api-access-6pj85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:37.632807 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.632694 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.632807 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.632789 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:37.632807 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.632787 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:37.633128 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.633104 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.633227 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.633210 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.633321 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.633298 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.635716 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.635696 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.642268 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.642248 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config" pod "393a4718-b786-4ac2-b4b1-cdb250c57261" (UID: "393a4718-b786-4ac2-b4b1-cdb250c57261"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:37.728740 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728713 6554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728740 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728739 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pj85\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-kube-api-access-6pj85\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728754 6554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-cluster-tls-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728766 6554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728781 6554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728794 6554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-config-out\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728808 6554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-main-db\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728820 6554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/393a4718-b786-4ac2-b4b1-cdb250c57261-tls-assets\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728833 6554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-main-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728847 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-metrics-client-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728862 6554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/393a4718-b786-4ac2-b4b1-cdb250c57261-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.728893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728876 6554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:37.729193 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:37.728915 6554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/393a4718-b786-4ac2-b4b1-cdb250c57261-web-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:38.304155 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304126 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" exitCode=0 Oct 30 14:48:38.304155 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304157 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" exitCode=0 Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304166 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" exitCode=0 Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304175 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" exitCode=0 Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304184 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" exitCode=0 Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304195 6554 generic.go:356] "Generic (PLEG): container finished" podID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" exitCode=0 Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304199 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304226 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304262 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304289 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304315 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304336 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304340 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304376 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} Oct 30 14:48:38.304490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.304399 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"393a4718-b786-4ac2-b4b1-cdb250c57261","Type":"ContainerDied","Data":"162b4a4f77cfd1e5e7c8ecaaa29cb8a2714d5de9528e2efc2ec7c360256a2b3d"} Oct 30 14:48:38.311571 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.311551 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.318126 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.318110 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.324724 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.324704 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.330907 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.330893 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.337597 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.337579 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.344197 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.344176 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.350748 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.350731 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.351002 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.350982 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.351052 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351010 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} err="failed to get container status \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" Oct 30 14:48:38.351052 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351038 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.351267 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.351251 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.351307 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351273 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} err="failed to get container status \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" Oct 30 14:48:38.351307 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351287 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.351507 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.351491 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.351555 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351511 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} err="failed to get container status \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" Oct 30 14:48:38.351555 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351524 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.351739 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.351721 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.351777 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351744 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} err="failed to get container status \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" Oct 30 14:48:38.351777 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351756 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.351977 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.351959 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.352045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.351982 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} err="failed to get container status \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" Oct 30 14:48:38.352045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352007 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.352249 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.352234 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.352283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352254 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} err="failed to get container status \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" Oct 30 14:48:38.352283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352268 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.352504 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:38.352487 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.352574 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352509 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} err="failed to get container status \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" Oct 30 14:48:38.352574 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352521 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.352756 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352740 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} err="failed to get container status \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" Oct 30 14:48:38.352812 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352756 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.352991 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352969 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} err="failed to get container status \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" Oct 30 14:48:38.353056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.352993 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.353270 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353250 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} err="failed to get container status \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" Oct 30 14:48:38.353270 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353268 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.353538 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353519 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} err="failed to get container status \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" Oct 30 14:48:38.353538 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353539 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.353787 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353767 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} err="failed to get container status \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" Oct 30 14:48:38.353787 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353787 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.353994 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353977 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} err="failed to get container status \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" Oct 30 14:48:38.354051 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.353993 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.354203 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354185 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} err="failed to get container status \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" Oct 30 14:48:38.354251 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354203 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.354421 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354404 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} err="failed to get container status \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" Oct 30 14:48:38.354472 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354422 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.354614 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354595 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} err="failed to get container status \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" Oct 30 14:48:38.354614 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354612 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.354834 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354818 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} err="failed to get container status \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" Oct 30 14:48:38.354834 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.354834 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.355048 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355030 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} err="failed to get container status \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" Oct 30 14:48:38.355095 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355048 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.355259 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355244 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} err="failed to get container status \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" Oct 30 14:48:38.355259 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355259 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.355473 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355457 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} err="failed to get container status \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" Oct 30 14:48:38.355516 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355472 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.355714 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355695 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} err="failed to get container status \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" Oct 30 14:48:38.355764 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355715 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.355924 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355905 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} err="failed to get container status \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" Oct 30 14:48:38.355924 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.355924 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.356147 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356131 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} err="failed to get container status \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" Oct 30 14:48:38.356147 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356147 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.356373 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356336 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} err="failed to get container status \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" Oct 30 14:48:38.356420 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356372 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.356581 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356565 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} err="failed to get container status \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" Oct 30 14:48:38.356650 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356582 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.356796 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356777 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} err="failed to get container status \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" Oct 30 14:48:38.356854 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.356796 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.357042 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357024 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} err="failed to get container status \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" Oct 30 14:48:38.357082 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357042 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.357247 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357231 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} err="failed to get container status \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" Oct 30 14:48:38.357289 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357246 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.357516 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357491 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} err="failed to get container status \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" Oct 30 14:48:38.357573 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357516 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.357719 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357703 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} err="failed to get container status \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" Oct 30 14:48:38.357767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357719 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.357922 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357906 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} err="failed to get container status \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" Oct 30 14:48:38.357970 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.357921 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.358143 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358123 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} err="failed to get container status \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" Oct 30 14:48:38.358143 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358142 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.358387 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358334 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} err="failed to get container status \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" Oct 30 14:48:38.358387 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358372 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.358676 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358660 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} err="failed to get container status \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" Oct 30 14:48:38.358715 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358675 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.358910 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358886 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} err="failed to get container status \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" Oct 30 14:48:38.358958 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.358911 6554 scope.go:122] "RemoveContainer" containerID="b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355" Oct 30 14:48:38.359115 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359099 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355"} err="failed to get container status \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": rpc error: code = NotFound desc = could not find container \"b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355\": container with ID starting with b652ba002a90b7d424ef27519aed17d0d25accdd28bf8852c3f0ec606ca5f355 not found: ID does not exist" Oct 30 14:48:38.359115 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359113 6554 scope.go:122] "RemoveContainer" containerID="1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369" Oct 30 14:48:38.359307 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359290 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369"} err="failed to get container status \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": rpc error: code = NotFound desc = could not find container \"1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369\": container with ID starting with 1306d55007badf19ee9424d88115c4f3431f46c9709cd6e1d80d35929d702369 not found: ID does not exist" Oct 30 14:48:38.359372 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359308 6554 scope.go:122] "RemoveContainer" containerID="d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664" Oct 30 14:48:38.359531 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359511 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664"} err="failed to get container status \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": rpc error: code = NotFound desc = could not find container \"d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664\": container with ID starting with d8c4bd05bc8ec1b49b9f9a1d9fca5a8a3f70b60c1f747e79807b8f66a41f1664 not found: ID does not exist" Oct 30 14:48:38.359575 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359531 6554 scope.go:122] "RemoveContainer" containerID="99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e" Oct 30 14:48:38.359740 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359724 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e"} err="failed to get container status \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": rpc error: code = NotFound desc = could not find container \"99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e\": container with ID starting with 99a8c4d9b362d0a9f14eebc704b8263a7653a95109c0ea21488b29967010457e not found: ID does not exist" Oct 30 14:48:38.359740 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359739 6554 scope.go:122] "RemoveContainer" containerID="980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8" Oct 30 14:48:38.359927 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359912 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8"} err="failed to get container status \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": rpc error: code = NotFound desc = could not find container \"980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8\": container with ID starting with 980e497f7f87d1b3585a81bbfe6b10865ec80f9e19ed187e8a66934818ab02e8 not found: ID does not exist" Oct 30 14:48:38.359968 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.359927 6554 scope.go:122] "RemoveContainer" containerID="b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80" Oct 30 14:48:38.360134 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.360117 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80"} err="failed to get container status \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": rpc error: code = NotFound desc = could not find container \"b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80\": container with ID starting with b3dd273c6ba1ce58714ec6f528a5e52b91f959bf784a2ce739293069dae1ba80 not found: ID does not exist" Oct 30 14:48:38.360177 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.360134 6554 scope.go:122] "RemoveContainer" containerID="f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173" Oct 30 14:48:38.360327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.360313 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173"} err="failed to get container status \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": rpc error: code = NotFound desc = could not find container \"f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173\": container with ID starting with f38a825f39de90a594a31889895dcad4dfdd931547a2d2203ecb42781ed88173 not found: ID does not exist" Oct 30 14:48:38.376713 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.376695 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:38.390132 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:38.390115 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 14:48:39.199745 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:48:39.199711 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:48:40.151422 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:40.151344 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" path="/var/lib/kubelet/pods/393a4718-b786-4ac2-b4b1-cdb250c57261/volumes" Oct 30 14:48:41.984819 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.984785 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:41.985426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.985156 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy" containerID="cri-o://69fa3f707c2c52ddca4de1c012377a567ad42ba2eb1bf5c3345a34d41538e3f0" gracePeriod=600 Oct 30 14:48:41.985426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.985208 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4fdaf243b2ba0e72e8e2210480f797faf17330681e8c0635db7a732609af7c9c" gracePeriod=600 Oct 30 14:48:41.985426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.985246 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-web" containerID="cri-o://5aa58ce3fee9d2dfbcbd8664ffd0ec3e1fac15d84b1669fcaf3880dd610e84ee" gracePeriod=600 Oct 30 14:48:41.985426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.985248 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="thanos-sidecar" containerID="cri-o://c1aae2d5db9089161023e0f1e98576bb74657274a17def109ac078cb7af80a8a" gracePeriod=600 Oct 30 14:48:41.985426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.985190 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="config-reloader" containerID="cri-o://5c997bb1c8f930ff2dc91bcb785a7a1dc18c9276f49fdf17dd924769a70b359b" gracePeriod=600 Oct 30 14:48:41.985426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:41.985131 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="prometheus" containerID="cri-o://6dde33c5de4f18713c381b4d5d0a5ecc767d771aee7b723fab08b68d69b8c998" gracePeriod=600 Oct 30 14:48:42.197257 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.197224 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-748747647-9rm5d" podUID="88a21bd0-b038-400e-8e85-be20c12f8ec0" containerName="console" containerID="cri-o://6bda8771326d8afa36c484da6dc5e2d19dba5f92ebaa84f1234855203eb6dc26" gracePeriod=15 Oct 30 14:48:42.202595 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.202560 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-56cf858bf9-pd4k9" podUID="80b69c00-6f7c-4048-aa22-c388091fdec2" containerName="console" containerID="cri-o://a806e1a938f025816ce8ac207f0f36e0a5641894aa00c7dfc14315bd03a914a5" gracePeriod=15 Oct 30 14:48:42.322656 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.322624 6554 generic.go:356] "Generic (PLEG): container finished" podID="80b69c00-6f7c-4048-aa22-c388091fdec2" containerID="a806e1a938f025816ce8ac207f0f36e0a5641894aa00c7dfc14315bd03a914a5" exitCode=0 Oct 30 14:48:42.322792 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.322737 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cf858bf9-pd4k9" event={"ID":"80b69c00-6f7c-4048-aa22-c388091fdec2","Type":"ContainerDied","Data":"a806e1a938f025816ce8ac207f0f36e0a5641894aa00c7dfc14315bd03a914a5"} Oct 30 14:48:42.326786 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326763 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="4fdaf243b2ba0e72e8e2210480f797faf17330681e8c0635db7a732609af7c9c" exitCode=0 Oct 30 14:48:42.326786 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326785 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="69fa3f707c2c52ddca4de1c012377a567ad42ba2eb1bf5c3345a34d41538e3f0" exitCode=0 Oct 30 14:48:42.326786 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326791 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="5aa58ce3fee9d2dfbcbd8664ffd0ec3e1fac15d84b1669fcaf3880dd610e84ee" exitCode=0 Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326797 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="c1aae2d5db9089161023e0f1e98576bb74657274a17def109ac078cb7af80a8a" exitCode=0 Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326803 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="5c997bb1c8f930ff2dc91bcb785a7a1dc18c9276f49fdf17dd924769a70b359b" exitCode=0 Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326810 6554 generic.go:356] "Generic (PLEG): container finished" podID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerID="6dde33c5de4f18713c381b4d5d0a5ecc767d771aee7b723fab08b68d69b8c998" exitCode=0 Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326838 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"4fdaf243b2ba0e72e8e2210480f797faf17330681e8c0635db7a732609af7c9c"} Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326901 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"69fa3f707c2c52ddca4de1c012377a567ad42ba2eb1bf5c3345a34d41538e3f0"} Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326941 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"5aa58ce3fee9d2dfbcbd8664ffd0ec3e1fac15d84b1669fcaf3880dd610e84ee"} Oct 30 14:48:42.327015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.326982 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"c1aae2d5db9089161023e0f1e98576bb74657274a17def109ac078cb7af80a8a"} Oct 30 14:48:42.327397 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.327022 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"5c997bb1c8f930ff2dc91bcb785a7a1dc18c9276f49fdf17dd924769a70b359b"} Oct 30 14:48:42.327397 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.327045 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:42.327397 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.327061 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"6dde33c5de4f18713c381b4d5d0a5ecc767d771aee7b723fab08b68d69b8c998"} Oct 30 14:48:42.327397 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.327101 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e36a4793-6b81-439f-b357-cc7e80803eb0","Type":"ContainerDied","Data":"09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2"} Oct 30 14:48:42.327397 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.327144 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09588104d7b37b9ffb65a8318f6eeb2db6ec52b66051005c24c2b82b6c8a1cf2" Oct 30 14:48:42.331950 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.329382 6554 generic.go:356] "Generic (PLEG): container finished" podID="88a21bd0-b038-400e-8e85-be20c12f8ec0" containerID="6bda8771326d8afa36c484da6dc5e2d19dba5f92ebaa84f1234855203eb6dc26" exitCode=0 Oct 30 14:48:42.331950 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.329457 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748747647-9rm5d" event={"ID":"88a21bd0-b038-400e-8e85-be20c12f8ec0","Type":"ContainerDied","Data":"6bda8771326d8afa36c484da6dc5e2d19dba5f92ebaa84f1234855203eb6dc26"} Oct 30 14:48:42.359829 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.359732 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.359829 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.359812 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.359988 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.359867 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.359988 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.359904 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.359988 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.359950 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360079 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360007 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360079 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360038 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360079 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360066 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360167 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360100 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360167 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360141 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360167 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360142 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.360304 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360174 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360304 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360202 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360304 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360241 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360304 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360272 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360527 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360324 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360527 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360386 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360527 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360416 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360527 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360453 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360527 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360483 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.360794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.360774 6554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.361551 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.361261 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.362126 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.362083 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.362833 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.362559 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.363906 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.363457 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.363906 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.363039 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.363906 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.363656 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.363906 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.363667 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:42.363906 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.363876 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.364220 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.363914 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.364873 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.364835 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.366017 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.365995 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "kube-api-access-4kpnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:42.366401 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.366138 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.366401 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.366313 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.366401 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.366326 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:42.366867 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.366841 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:42.367787 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.367743 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.367911 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.367881 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.382955 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.382930 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.441728 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.441701 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:42.461111 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461084 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461133 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461162 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461195 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461292 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461341 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461387 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461441 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461416 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") pod \"88a21bd0-b038-400e-8e85-be20c12f8ec0\" (UID: \"88a21bd0-b038-400e-8e85-be20c12f8ec0\") " Oct 30 14:48:42.461651 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461445 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") pod \"e36a4793-6b81-439f-b357-cc7e80803eb0\" (UID: \"e36a4793-6b81-439f-b357-cc7e80803eb0\") " Oct 30 14:48:42.461651 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461594 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:42.461751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461640 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.461751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461711 6554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-web-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461731 6554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-tls-assets\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461748 6554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461763 6554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-metrics-client-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461799 6554 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88a21bd0-b038-400e-8e85-be20c12f8ec0-tmp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461813 6554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-db\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461829 6554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-metrics-client-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461842 6554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e36a4793-6b81-439f-b357-cc7e80803eb0-config-out\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461856 6554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461869 6554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461882 6554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461897 6554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461895 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1" pod "e36a4793-6b81-439f-b357-cc7e80803eb0" (UID: "e36a4793-6b81-439f-b357-cc7e80803eb0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461911 6554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461926 6554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461941 6554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461955 6554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.461963 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461968 6554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-kube-rbac-proxy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.462643 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461982 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kpnl\" (UniqueName: \"kubernetes.io/projected/e36a4793-6b81-439f-b357-cc7e80803eb0-kube-api-access-4kpnl\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.462643 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.461995 6554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e36a4793-6b81-439f-b357-cc7e80803eb0-secret-grpc-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.462643 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.462076 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.462643 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.462176 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.462643 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.462218 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.463831 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.463809 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "kube-api-access-df7bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:42.464067 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.464029 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.464301 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.464273 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config" pod "88a21bd0-b038-400e-8e85-be20c12f8ec0" (UID: "88a21bd0-b038-400e-8e85-be20c12f8ec0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.475603 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.475584 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:42.562829 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562769 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.562829 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562803 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563000 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562833 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563000 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562861 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563000 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562900 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563000 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562940 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563000 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.562969 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563246 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563000 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") pod \"80b69c00-6f7c-4048-aa22-c388091fdec2\" (UID: \"80b69c00-6f7c-4048-aa22-c388091fdec2\") " Oct 30 14:48:42.563246 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563079 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563265 6554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-trusted-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563285 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df7bw\" (UniqueName: \"kubernetes.io/projected/88a21bd0-b038-400e-8e85-be20c12f8ec0-kube-api-access-df7bw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563300 6554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563314 6554 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80b69c00-6f7c-4048-aa22-c388091fdec2-tmp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563326 6554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-oauth-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563341 6554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-oauth-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563388 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563377 6554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/88a21bd0-b038-400e-8e85-be20c12f8ec0-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563392 6554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e36a4793-6b81-439f-b357-cc7e80803eb0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563407 6554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/88a21bd0-b038-400e-8e85-be20c12f8ec0-console-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.563743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563388 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.563743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563413 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.563743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563427 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.563743 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.563698 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:48:42.565012 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.564989 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.565095 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.565022 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "kube-api-access-kgrpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:48:42.565095 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.565057 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert" pod "80b69c00-6f7c-4048-aa22-c388091fdec2" (UID: "80b69c00-6f7c-4048-aa22-c388091fdec2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:48:42.664640 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664617 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgrpk\" (UniqueName: \"kubernetes.io/projected/80b69c00-6f7c-4048-aa22-c388091fdec2-kube-api-access-kgrpk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.664640 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664639 6554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.664749 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664649 6554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80b69c00-6f7c-4048-aa22-c388091fdec2-console-oauth-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.664749 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664658 6554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-oauth-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.664749 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664667 6554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.664749 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664675 6554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-console-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:42.664749 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:42.664683 6554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80b69c00-6f7c-4048-aa22-c388091fdec2-trusted-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:48:43.335283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.335217 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56cf858bf9-pd4k9" Oct 30 14:48:43.335283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.335240 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56cf858bf9-pd4k9" event={"ID":"80b69c00-6f7c-4048-aa22-c388091fdec2","Type":"ContainerDied","Data":"e978785d06068888f81b7c37a56918924df9f82350d0868966325cb916d70e83"} Oct 30 14:48:43.335789 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.335307 6554 scope.go:122] "RemoveContainer" containerID="a806e1a938f025816ce8ac207f0f36e0a5641894aa00c7dfc14315bd03a914a5" Oct 30 14:48:43.337032 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.337007 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748747647-9rm5d" Oct 30 14:48:43.337187 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.337059 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748747647-9rm5d" event={"ID":"88a21bd0-b038-400e-8e85-be20c12f8ec0","Type":"ContainerDied","Data":"cc606ad47b1e3eb8fac7c0048df1e1c030164682997ece789a6240b6e70b434d"} Oct 30 14:48:43.337260 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.337194 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.345993 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.345974 6554 scope.go:122] "RemoveContainer" containerID="6bda8771326d8afa36c484da6dc5e2d19dba5f92ebaa84f1234855203eb6dc26" Oct 30 14:48:43.370783 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.370765 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:43.371022 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.371008 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:43.375994 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.375973 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:43.401366 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.401323 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748747647-9rm5d"] Oct 30 14:48:43.405623 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.405603 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-748747647-9rm5d"] Oct 30 14:48:43.463582 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.463559 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:43.477284 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.477263 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:43.507516 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.507495 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56cf858bf9-pd4k9"] Oct 30 14:48:43.514696 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.514669 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:43.515219 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515201 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="alertmanager" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515222 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="alertmanager" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515256 6554 cpu_manager.go:414] "containerMap: removing container" podUID="80b69c00-6f7c-4048-aa22-c388091fdec2" containerName="console" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515263 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b69c00-6f7c-4048-aa22-c388091fdec2" containerName="console" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515282 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="init-config-reloader" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515290 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="init-config-reloader" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515304 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="prom-label-proxy" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515311 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="prom-label-proxy" Oct 30 14:48:43.515326 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515329 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515336 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515363 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="prometheus" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515370 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="prometheus" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515385 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="init-config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515392 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="init-config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515404 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-thanos" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515411 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-thanos" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515427 6554 cpu_manager.go:414] "containerMap: removing container" podUID="88a21bd0-b038-400e-8e85-be20c12f8ec0" containerName="console" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515434 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a21bd0-b038-400e-8e85-be20c12f8ec0" containerName="console" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515459 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515465 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515475 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-web" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515487 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-web" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515496 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-metric" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515503 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-metric" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515514 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515521 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515534 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="thanos-sidecar" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515541 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="thanos-sidecar" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515552 6554 cpu_manager.go:414] "containerMap: removing container" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515559 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515577 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-web" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515583 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-web" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515700 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b69c00-6f7c-4048-aa22-c388091fdec2" containerName="console" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515711 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="init-config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515726 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="prom-label-proxy" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515734 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515742 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="prometheus" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515753 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="init-config-reloader" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515768 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-thanos" Oct 30 14:48:43.515767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515777 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a21bd0-b038-400e-8e85-be20c12f8ec0" containerName="console" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515794 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515804 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-web" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515812 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="kube-rbac-proxy-metric" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515827 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515834 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="thanos-sidecar" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515848 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="config-reloader" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515856 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" containerName="kube-rbac-proxy-web" Oct 30 14:48:43.517626 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.515866 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="393a4718-b786-4ac2-b4b1-cdb250c57261" containerName="alertmanager" Oct 30 14:48:43.522304 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.522285 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56cf858bf9-pd4k9"] Oct 30 14:48:43.522403 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.522391 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.554518 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.554502 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 14:48:43.555187 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555172 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 14:48:43.555508 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555485 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ctc47fv49jduf\"" Oct 30 14:48:43.555604 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555515 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tcq4s\"" Oct 30 14:48:43.555604 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555552 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 14:48:43.555604 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555573 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 14:48:43.555723 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555617 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 14:48:43.555723 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555647 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 14:48:43.555723 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555669 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 14:48:43.555816 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.555755 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 14:48:43.556089 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.556072 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 14:48:43.559118 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.559100 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 14:48:43.559202 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.559157 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 14:48:43.570130 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.570113 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:43.570948 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.570931 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 14:48:43.571059 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.571019 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 14:48:43.572220 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572202 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572339 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572227 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572339 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572244 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cf0c34-2548-40ca-81df-b06ca85f0055-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572339 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572261 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572456 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572398 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572456 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572444 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cf0c34-2548-40ca-81df-b06ca85f0055-config-out\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572520 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572474 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572581 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572560 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-web-config\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572615 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572596 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572703 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572687 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-config\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572737 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572723 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572749 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmq9\" (UniqueName: \"kubernetes.io/projected/b6cf0c34-2548-40ca-81df-b06ca85f0055-kube-api-access-xwmq9\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572801 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572777 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572880 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572866 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572910 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572890 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572950 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572907 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.572992 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.572963 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.573031 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.573007 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.573071 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.573034 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.573071 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.573049 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.652403 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.652387 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 14:48:43.673763 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673743 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.673850 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673773 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.673850 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673790 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.673850 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673818 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.673850 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673843 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673872 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673894 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673923 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673943 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673966 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cf0c34-2548-40ca-81df-b06ca85f0055-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.673987 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674006 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674040 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cf0c34-2548-40ca-81df-b06ca85f0055-config-out\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674063 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674090 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-web-config\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674115 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674156 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-config\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674185 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674209 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmq9\" (UniqueName: \"kubernetes.io/projected/b6cf0c34-2548-40ca-81df-b06ca85f0055-kube-api-access-xwmq9\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674239 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674838 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674604 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674838 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674670 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.674838 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.674702 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.675220 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.675191 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.675456 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.675428 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.675888 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.675652 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.676365 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.676108 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.677250 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.677204 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b6cf0c34-2548-40ca-81df-b06ca85f0055-config-out\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.677769 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.677740 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.678263 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.678233 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-config\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.678401 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.678371 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b6cf0c34-2548-40ca-81df-b06ca85f0055-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.679411 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.679309 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-web-config\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.679411 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.679370 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.679661 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.679637 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.679731 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.679695 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b6cf0c34-2548-40ca-81df-b06ca85f0055-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.679794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.679753 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.680046 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.680026 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.680166 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.680149 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.680750 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.680725 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b6cf0c34-2548-40ca-81df-b06ca85f0055-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.699661 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.699641 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmq9\" (UniqueName: \"kubernetes.io/projected/b6cf0c34-2548-40ca-81df-b06ca85f0055-kube-api-access-xwmq9\") pod \"prometheus-k8s-0\" (UID: \"b6cf0c34-2548-40ca-81df-b06ca85f0055\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:43.845755 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:43.845732 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:44.151611 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.151580 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b69c00-6f7c-4048-aa22-c388091fdec2" path="/var/lib/kubelet/pods/80b69c00-6f7c-4048-aa22-c388091fdec2/volumes" Oct 30 14:48:44.151992 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.151980 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a21bd0-b038-400e-8e85-be20c12f8ec0" path="/var/lib/kubelet/pods/88a21bd0-b038-400e-8e85-be20c12f8ec0/volumes" Oct 30 14:48:44.152341 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.152330 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e36a4793-6b81-439f-b357-cc7e80803eb0" path="/var/lib/kubelet/pods/e36a4793-6b81-439f-b357-cc7e80803eb0/volumes" Oct 30 14:48:44.185523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.185505 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 14:48:44.187476 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:48:44.187448 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cf0c34_2548_40ca_81df_b06ca85f0055.slice/crio-a1c7a1fd06f595af6904f34f23196ae0259c4b1caa4e919c51ea51983d471cf2 WatchSource:0}: Error finding container a1c7a1fd06f595af6904f34f23196ae0259c4b1caa4e919c51ea51983d471cf2: Status 404 returned error can't find the container with id a1c7a1fd06f595af6904f34f23196ae0259c4b1caa4e919c51ea51983d471cf2 Oct 30 14:48:44.342697 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.342672 6554 generic.go:356] "Generic (PLEG): container finished" podID="b6cf0c34-2548-40ca-81df-b06ca85f0055" containerID="697baa1fc9e8adb86b794e971127526b3010bc6cd528d8dcccefeef39134efc2" exitCode=0 Oct 30 14:48:44.343051 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.342760 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerDied","Data":"697baa1fc9e8adb86b794e971127526b3010bc6cd528d8dcccefeef39134efc2"} Oct 30 14:48:44.343051 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.342852 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"a1c7a1fd06f595af6904f34f23196ae0259c4b1caa4e919c51ea51983d471cf2"} Oct 30 14:48:44.348550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.348533 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5dcc966694-t2bvg" Oct 30 14:48:44.503597 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:44.503577 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5c666f7f-p2trj"] Oct 30 14:48:45.353291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.353222 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"a07c4f0368f49c41913994b4a9325963d212386c95a7a9b1b25ddc520ab37650"} Oct 30 14:48:45.353691 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.353295 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"c5ff53f82fdcb3e7e94b62884dc25c0c95e50d26899539df3960483482410a8c"} Oct 30 14:48:45.353691 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.353325 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"d3dcd6773938d5a9703ce7324ed8db263bad09db1d1363530cf665c153384d56"} Oct 30 14:48:45.353691 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.353362 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"bf1c2144d3d1681cc8289af17f0acd4f1ff060cf98b4220937f3e696a08094bf"} Oct 30 14:48:45.353691 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.353393 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"efbf5593704ee6fc8db0bc4357e249905857b8c3132b4acce3e44c3f9d6ce7a5"} Oct 30 14:48:45.353691 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.353419 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b6cf0c34-2548-40ca-81df-b06ca85f0055","Type":"ContainerStarted","Data":"07efd90de92bc8e5f8c70327e6b63604afef21d571c5520876ed5086e72bebcb"} Oct 30 14:48:45.430739 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:45.430693 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.414341952 podStartE2EDuration="2.430676543s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.334591ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:48:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:48:45.429371897 +0000 UTC m=+29.388413310" watchObservedRunningTime="2026-10-30 14:48:45.430676543 +0000 UTC m=+29.389717949" Oct 30 14:48:48.846820 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:48.846782 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:48:50.568520 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:48:50.568484 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fnbqw" Oct 30 14:49:09.202065 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:49:09.202012 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:49:09.524105 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.524002 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b5c666f7f-p2trj" podUID="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" containerName="console" containerID="cri-o://8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f" gracePeriod=15 Oct 30 14:49:09.776585 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.776524 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:49:09.861650 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861613 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.861760 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861673 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.861819 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861799 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.861863 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861849 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.861909 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861873 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.861909 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861884 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 14:49:09.861909 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861894 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.862057 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861935 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.862057 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861968 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:09.862057 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.861980 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") pod \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\" (UID: \"a70892a8-ba07-4d35-b23a-6884e1b7d1a5\") " Oct 30 14:49:09.862271 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862249 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:09.862378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862293 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:09.862378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862323 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 14:49:09.862472 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862430 6554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.862472 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862452 6554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.862472 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862466 6554 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-tmp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.862663 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.862477 6554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-trusted-ca-bundle\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.864027 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.863999 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "kube-api-access-rmqf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:49:09.864126 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.864103 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:09.864175 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.864157 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert" pod "a70892a8-ba07-4d35-b23a-6884e1b7d1a5" (UID: "a70892a8-ba07-4d35-b23a-6884e1b7d1a5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:49:09.963580 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.963554 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmqf7\" (UniqueName: \"kubernetes.io/projected/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-kube-api-access-rmqf7\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.963580 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.963577 6554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-oauth-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.963697 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.963586 6554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-serving-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:09.963697 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:09.963595 6554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a70892a8-ba07-4d35-b23a-6884e1b7d1a5-console-oauth-config\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:49:10.453790 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.453724 6554 generic.go:356] "Generic (PLEG): container finished" podID="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" containerID="8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f" exitCode=0 Oct 30 14:49:10.453790 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.453781 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b5c666f7f-p2trj" Oct 30 14:49:10.454181 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.453784 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c666f7f-p2trj" event={"ID":"a70892a8-ba07-4d35-b23a-6884e1b7d1a5","Type":"ContainerDied","Data":"8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f"} Oct 30 14:49:10.454181 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.453915 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b5c666f7f-p2trj" event={"ID":"a70892a8-ba07-4d35-b23a-6884e1b7d1a5","Type":"ContainerDied","Data":"b4feb442d6e2eb9beae314dd17589722b65ec472b74af243728a793e32996720"} Oct 30 14:49:10.454181 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.453948 6554 scope.go:122] "RemoveContainer" containerID="8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f" Oct 30 14:49:10.462848 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.462829 6554 scope.go:122] "RemoveContainer" containerID="8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f" Oct 30 14:49:10.463147 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:49:10.463125 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f\": container with ID starting with 8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f not found: ID does not exist" containerID="8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f" Oct 30 14:49:10.463194 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.463156 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f"} err="failed to get container status \"8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f\": rpc error: code = NotFound desc = could not find container \"8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f\": container with ID starting with 8f27f3be8180efd731dc36fb60412f13ac7bd133a0c6f9a6abdc44f31cf82b2f not found: ID does not exist" Oct 30 14:49:10.510047 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.510019 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b5c666f7f-p2trj"] Oct 30 14:49:10.523726 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:10.523701 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b5c666f7f-p2trj"] Oct 30 14:49:12.154316 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:12.154270 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" path="/var/lib/kubelet/pods/a70892a8-ba07-4d35-b23a-6884e1b7d1a5/volumes" Oct 30 14:49:39.023699 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:39.023665 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:49:39.039246 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:39.039220 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:49:39.200384 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:49:39.200339 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:49:39.574290 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:49:39.574265 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 14:50:09.200776 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:50:09.200692 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:50:39.199834 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:50:39.199791 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:51:09.200083 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:51:09.200048 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:51:12.802601 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.802552 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-qbn88"] Oct 30 14:51:12.803032 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.802881 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" containerName="console" Oct 30 14:51:12.803032 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.802897 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" containerName="console" Oct 30 14:51:12.803032 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.802963 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a70892a8-ba07-4d35-b23a-6884e1b7d1a5" containerName="console" Oct 30 14:51:12.806018 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.805997 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:12.839307 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.839277 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 14:51:12.839468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.839372 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k8vvm\"" Oct 30 14:51:12.839468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.839394 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 14:51:12.840003 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.839984 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-qbn88"] Oct 30 14:51:12.922168 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.922141 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpc59\" (UniqueName: \"kubernetes.io/projected/ac0ca8d4-1e3e-410b-8340-d5033539f5bf-kube-api-access-tpc59\") pod \"openshift-lws-operator-8465994787-qbn88\" (UID: \"ac0ca8d4-1e3e-410b-8340-d5033539f5bf\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:12.922278 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:12.922195 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac0ca8d4-1e3e-410b-8340-d5033539f5bf-tmp\") pod \"openshift-lws-operator-8465994787-qbn88\" (UID: \"ac0ca8d4-1e3e-410b-8340-d5033539f5bf\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:13.023468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.023444 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac0ca8d4-1e3e-410b-8340-d5033539f5bf-tmp\") pod \"openshift-lws-operator-8465994787-qbn88\" (UID: \"ac0ca8d4-1e3e-410b-8340-d5033539f5bf\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:13.023582 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.023494 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpc59\" (UniqueName: \"kubernetes.io/projected/ac0ca8d4-1e3e-410b-8340-d5033539f5bf-kube-api-access-tpc59\") pod \"openshift-lws-operator-8465994787-qbn88\" (UID: \"ac0ca8d4-1e3e-410b-8340-d5033539f5bf\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:13.023789 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.023772 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac0ca8d4-1e3e-410b-8340-d5033539f5bf-tmp\") pod \"openshift-lws-operator-8465994787-qbn88\" (UID: \"ac0ca8d4-1e3e-410b-8340-d5033539f5bf\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:13.048362 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.048334 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpc59\" (UniqueName: \"kubernetes.io/projected/ac0ca8d4-1e3e-410b-8340-d5033539f5bf-kube-api-access-tpc59\") pod \"openshift-lws-operator-8465994787-qbn88\" (UID: \"ac0ca8d4-1e3e-410b-8340-d5033539f5bf\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:13.137427 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.137408 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" Oct 30 14:51:13.434747 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.434690 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-qbn88"] Oct 30 14:51:13.438021 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:51:13.437997 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac0ca8d4_1e3e_410b_8340_d5033539f5bf.slice/crio-0e172ce49508a411871f5e2d574576529817838a2b46b4926eded726938ee5f7 WatchSource:0}: Error finding container 0e172ce49508a411871f5e2d574576529817838a2b46b4926eded726938ee5f7: Status 404 returned error can't find the container with id 0e172ce49508a411871f5e2d574576529817838a2b46b4926eded726938ee5f7 Oct 30 14:51:13.439557 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.439540 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:51:13.908001 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:13.907971 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" event={"ID":"ac0ca8d4-1e3e-410b-8340-d5033539f5bf","Type":"ContainerStarted","Data":"0e172ce49508a411871f5e2d574576529817838a2b46b4926eded726938ee5f7"} Oct 30 14:51:16.920796 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:16.920757 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" event={"ID":"ac0ca8d4-1e3e-410b-8340-d5033539f5bf","Type":"ContainerStarted","Data":"7d148ffadabb9c95a462762ac108196fbd5823a1c97b2162b34210dc5161bc95"} Oct 30 14:51:16.963997 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:16.963951 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-qbn88" podStartSLOduration=2.47468065 podStartE2EDuration="4.963937331s" totalImagesPullingTime="2.489256681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:16.963161146 +0000 UTC m=+180.922202550" watchObservedRunningTime="2026-10-30 14:51:16.963937331 +0000 UTC m=+180.922978729" Oct 30 14:51:32.444599 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.444522 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw"] Oct 30 14:51:32.451284 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.451265 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.521061 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.521034 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw"] Oct 30 14:51:32.559622 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.559597 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 14:51:32.559929 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.559908 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 14:51:32.560070 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.560058 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 14:51:32.560110 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.560075 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b8vcv\"" Oct 30 14:51:32.563468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.563449 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8cc79f79-0d0c-4670-9921-101d339eb468-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.563550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.563479 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cc79f79-0d0c-4670-9921-101d339eb468-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.563550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.563504 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.563550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.563527 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.563727 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.563705 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.563785 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.563761 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxfv4\" (UniqueName: \"kubernetes.io/projected/8cc79f79-0d0c-4670-9921-101d339eb468-kube-api-access-bxfv4\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.579980 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.579960 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 14:51:32.580065 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.580057 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 14:51:32.665039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665014 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.665159 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665079 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.665159 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665125 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxfv4\" (UniqueName: \"kubernetes.io/projected/8cc79f79-0d0c-4670-9921-101d339eb468-kube-api-access-bxfv4\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.665159 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665150 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8cc79f79-0d0c-4670-9921-101d339eb468-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.665323 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665178 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cc79f79-0d0c-4670-9921-101d339eb468-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.665323 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665218 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.665627 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.665600 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8cc79f79-0d0c-4670-9921-101d339eb468-manifests\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.667641 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.667614 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8cc79f79-0d0c-4670-9921-101d339eb468-tmp\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.667641 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.667631 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-metrics-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.667755 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.667728 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.667825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.667805 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cc79f79-0d0c-4670-9921-101d339eb468-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.726401 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.726328 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxfv4\" (UniqueName: \"kubernetes.io/projected/8cc79f79-0d0c-4670-9921-101d339eb468-kube-api-access-bxfv4\") pod \"opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw\" (UID: \"8cc79f79-0d0c-4670-9921-101d339eb468\") " pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.766244 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.766200 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:32.971559 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.971527 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-785f85dc95-898td"] Oct 30 14:51:32.975521 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:32.975501 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.001940 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.001917 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-m5kr4\"" Oct 30 14:51:33.002825 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.002808 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 14:51:33.002908 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.002895 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 14:51:33.003581 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.003567 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 14:51:33.009472 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.009453 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-785f85dc95-898td"] Oct 30 14:51:33.069005 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.068983 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/07be7af0-7cea-446e-a46d-814baf6907bf-manager-config\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.069105 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.069019 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07be7af0-7cea-446e-a46d-814baf6907bf-cert\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.069105 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.069037 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/07be7af0-7cea-446e-a46d-814baf6907bf-metrics-cert\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.069210 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.069192 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvkl7\" (UniqueName: \"kubernetes.io/projected/07be7af0-7cea-446e-a46d-814baf6907bf-kube-api-access-dvkl7\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.148468 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:51:33.148444 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc79f79_0d0c_4670_9921_101d339eb468.slice/crio-527522bcaf6164ee72b4a9c9c8fcc145ae371df766412df3037081d00c6f105b WatchSource:0}: Error finding container 527522bcaf6164ee72b4a9c9c8fcc145ae371df766412df3037081d00c6f105b: Status 404 returned error can't find the container with id 527522bcaf6164ee72b4a9c9c8fcc145ae371df766412df3037081d00c6f105b Oct 30 14:51:33.154320 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.154296 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw"] Oct 30 14:51:33.170561 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.170535 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/07be7af0-7cea-446e-a46d-814baf6907bf-manager-config\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.170644 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.170579 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07be7af0-7cea-446e-a46d-814baf6907bf-cert\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.170644 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.170597 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/07be7af0-7cea-446e-a46d-814baf6907bf-metrics-cert\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.170644 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.170634 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvkl7\" (UniqueName: \"kubernetes.io/projected/07be7af0-7cea-446e-a46d-814baf6907bf-kube-api-access-dvkl7\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.171114 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.171083 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/07be7af0-7cea-446e-a46d-814baf6907bf-manager-config\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.173099 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.173077 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/07be7af0-7cea-446e-a46d-814baf6907bf-cert\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.173099 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.173090 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/07be7af0-7cea-446e-a46d-814baf6907bf-metrics-cert\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.197883 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.197864 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvkl7\" (UniqueName: \"kubernetes.io/projected/07be7af0-7cea-446e-a46d-814baf6907bf-kube-api-access-dvkl7\") pod \"lws-controller-manager-785f85dc95-898td\" (UID: \"07be7af0-7cea-446e-a46d-814baf6907bf\") " pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.288156 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.287798 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:33.602945 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.602874 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-785f85dc95-898td"] Oct 30 14:51:33.606419 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:51:33.606394 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07be7af0_7cea_446e_a46d_814baf6907bf.slice/crio-a8d3917bcfdb801ceb165d8abdd53df4cb8868743d5456db8e2c8a3033316cec WatchSource:0}: Error finding container a8d3917bcfdb801ceb165d8abdd53df4cb8868743d5456db8e2c8a3033316cec: Status 404 returned error can't find the container with id a8d3917bcfdb801ceb165d8abdd53df4cb8868743d5456db8e2c8a3033316cec Oct 30 14:51:33.999502 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:33.999466 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" event={"ID":"07be7af0-7cea-446e-a46d-814baf6907bf","Type":"ContainerStarted","Data":"a8d3917bcfdb801ceb165d8abdd53df4cb8868743d5456db8e2c8a3033316cec"} Oct 30 14:51:34.001199 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:34.001155 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" event={"ID":"8cc79f79-0d0c-4670-9921-101d339eb468","Type":"ContainerStarted","Data":"527522bcaf6164ee72b4a9c9c8fcc145ae371df766412df3037081d00c6f105b"} Oct 30 14:51:37.014883 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:37.014818 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" event={"ID":"07be7af0-7cea-446e-a46d-814baf6907bf","Type":"ContainerStarted","Data":"48bca80a8fe14aa35f19cbb8d09aed58cf2576d92de8d4844837b6fae4313720"} Oct 30 14:51:37.015288 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:37.014960 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:37.016506 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:37.016479 6554 generic.go:356] "Generic (PLEG): container finished" podID="8cc79f79-0d0c-4670-9921-101d339eb468" containerID="23a97248168f9c2fc0074266e7d4ba9af9b984a0a845b9691cb3aabaf9dd5f65" exitCode=0 Oct 30 14:51:37.016603 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:37.016556 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" event={"ID":"8cc79f79-0d0c-4670-9921-101d339eb468","Type":"ContainerDied","Data":"23a97248168f9c2fc0074266e7d4ba9af9b984a0a845b9691cb3aabaf9dd5f65"} Oct 30 14:51:37.059299 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:37.059257 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" podStartSLOduration=2.601798385 podStartE2EDuration="5.059247001s" totalImagesPullingTime="2.457448616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:37.058395826 +0000 UTC m=+201.017437233" watchObservedRunningTime="2026-10-30 14:51:37.059247001 +0000 UTC m=+201.018288461" Oct 30 14:51:38.027813 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:38.027763 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" event={"ID":"8cc79f79-0d0c-4670-9921-101d339eb468","Type":"ContainerStarted","Data":"425697fcc5859826cf8299ea69aefe055d0063e75fcd9413c62af82384e87995"} Oct 30 14:51:38.028278 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:38.028154 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:38.125283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:38.125241 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" podStartSLOduration=2.208914262 podStartE2EDuration="6.125230779s" totalImagesPullingTime="3.345368043s" totalInitContainerRuntime="570.948474ms" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:38.11136551 +0000 UTC m=+202.070406905" watchObservedRunningTime="2026-10-30 14:51:38.125230779 +0000 UTC m=+202.084272184" Oct 30 14:51:39.199508 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:51:39.199461 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:51:48.030648 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:48.030611 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-785f85dc95-898td" Oct 30 14:51:49.034514 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:49.034486 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw" Oct 30 14:51:53.037846 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.037813 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99"] Oct 30 14:51:53.044649 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.044629 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.063757 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.063731 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 14:51:53.063866 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.063800 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 14:51:53.074038 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.074014 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-r5tgf\"" Oct 30 14:51:53.077330 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.077313 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 14:51:53.077567 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.077549 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 14:51:53.078290 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.078270 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99"] Oct 30 14:51:53.216162 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.216138 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad6de2a-89da-437a-9267-28357e2f8c77-tmp\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.216258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.216173 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkzh\" (UniqueName: \"kubernetes.io/projected/dad6de2a-89da-437a-9267-28357e2f8c77-kube-api-access-jmkzh\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.216258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.216194 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad6de2a-89da-437a-9267-28357e2f8c77-tls-certs\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.317190 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.317137 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad6de2a-89da-437a-9267-28357e2f8c77-tmp\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.317287 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.317193 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkzh\" (UniqueName: \"kubernetes.io/projected/dad6de2a-89da-437a-9267-28357e2f8c77-kube-api-access-jmkzh\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.317287 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.317224 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad6de2a-89da-437a-9267-28357e2f8c77-tls-certs\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.319435 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.319409 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dad6de2a-89da-437a-9267-28357e2f8c77-tmp\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.319713 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.319692 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dad6de2a-89da-437a-9267-28357e2f8c77-tls-certs\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.342913 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.342891 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkzh\" (UniqueName: \"kubernetes.io/projected/dad6de2a-89da-437a-9267-28357e2f8c77-kube-api-access-jmkzh\") pod \"kube-auth-proxy-6d8f97cc56-pzc99\" (UID: \"dad6de2a-89da-437a-9267-28357e2f8c77\") " pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.356706 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.356689 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" Oct 30 14:51:53.659604 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:53.659524 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99"] Oct 30 14:51:53.663501 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:51:53.663477 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad6de2a_89da_437a_9267_28357e2f8c77.slice/crio-43a33af3a1ffb6640a0aeabad0d53cdaf11f4e794d4aa062b0bac3844eba94ab WatchSource:0}: Error finding container 43a33af3a1ffb6640a0aeabad0d53cdaf11f4e794d4aa062b0bac3844eba94ab: Status 404 returned error can't find the container with id 43a33af3a1ffb6640a0aeabad0d53cdaf11f4e794d4aa062b0bac3844eba94ab Oct 30 14:51:54.100030 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:54.099961 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" event={"ID":"dad6de2a-89da-437a-9267-28357e2f8c77","Type":"ContainerStarted","Data":"43a33af3a1ffb6640a0aeabad0d53cdaf11f4e794d4aa062b0bac3844eba94ab"} Oct 30 14:51:56.109946 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:56.109922 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" event={"ID":"dad6de2a-89da-437a-9267-28357e2f8c77","Type":"ContainerStarted","Data":"64ffe5b0a8340965c6d93c40f06d57f20b80eadc509c146a71dfd4bd169c440c"} Oct 30 14:51:56.158762 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:51:56.158714 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d8f97cc56-pzc99" podStartSLOduration=0.818886031 podStartE2EDuration="3.158698114s" totalImagesPullingTime="2.339812083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:51:56.1577216 +0000 UTC m=+220.116763018" watchObservedRunningTime="2026-10-30 14:51:56.158698114 +0000 UTC m=+220.117739518" Oct 30 14:52:09.201066 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:52:09.201022 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:52:39.201757 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:52:39.201709 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:53:00.092461 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.092251 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-r5956"] Oct 30 14:53:00.095721 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.095702 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.121938 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.121907 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-r5956"] Oct 30 14:53:00.160432 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.160410 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 14:53:00.160432 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.160418 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 14:53:00.160580 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.160449 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6blcv\"" Oct 30 14:53:00.195987 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.195961 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88e05d5-b66e-4737-b87b-19fd58909603-metrics-certs\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.196098 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.196027 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88e05d5-b66e-4737-b87b-19fd58909603-cert\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.196188 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.196163 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2mc\" (UniqueName: \"kubernetes.io/projected/e88e05d5-b66e-4737-b87b-19fd58909603-kube-api-access-jm2mc\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.296712 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.296684 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88e05d5-b66e-4737-b87b-19fd58909603-cert\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.296848 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.296789 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2mc\" (UniqueName: \"kubernetes.io/projected/e88e05d5-b66e-4737-b87b-19fd58909603-kube-api-access-jm2mc\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.296926 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.296846 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88e05d5-b66e-4737-b87b-19fd58909603-metrics-certs\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.299264 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.299241 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e88e05d5-b66e-4737-b87b-19fd58909603-cert\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.299373 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.299241 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e88e05d5-b66e-4737-b87b-19fd58909603-metrics-certs\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.325038 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.324977 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2mc\" (UniqueName: \"kubernetes.io/projected/e88e05d5-b66e-4737-b87b-19fd58909603-kube-api-access-jm2mc\") pod \"odh-model-controller-f87fcf56c-r5956\" (UID: \"e88e05d5-b66e-4737-b87b-19fd58909603\") " pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.408226 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.408198 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:00.704708 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:00.704644 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-r5956"] Oct 30 14:53:00.707654 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:53:00.707631 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88e05d5_b66e_4737_b87b_19fd58909603.slice/crio-be4055ff26f207c2575036422d4c3d712cd9fcee3f5672a8d042a9ad7a27baa0 WatchSource:0}: Error finding container be4055ff26f207c2575036422d4c3d712cd9fcee3f5672a8d042a9ad7a27baa0: Status 404 returned error can't find the container with id be4055ff26f207c2575036422d4c3d712cd9fcee3f5672a8d042a9ad7a27baa0 Oct 30 14:53:01.385428 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:01.385389 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" event={"ID":"e88e05d5-b66e-4737-b87b-19fd58909603","Type":"ContainerStarted","Data":"be4055ff26f207c2575036422d4c3d712cd9fcee3f5672a8d042a9ad7a27baa0"} Oct 30 14:53:03.398790 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:03.398754 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" event={"ID":"e88e05d5-b66e-4737-b87b-19fd58909603","Type":"ContainerStarted","Data":"3cb59e69b15e11cbdab4b88dbe02a67eabb2e7f21eee609daf566006934d0a60"} Oct 30 14:53:03.399161 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:03.398894 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:03.441839 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:03.441745 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" podStartSLOduration=0.973126383 podStartE2EDuration="3.4417301s" totalImagesPullingTime="2.468603717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:53:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:53:03.441138575 +0000 UTC m=+287.400180022" watchObservedRunningTime="2026-10-30 14:53:03.4417301 +0000 UTC m=+287.400771506" Oct 30 14:53:09.201061 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:53:09.201017 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:53:14.405573 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:53:14.405541 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-r5956" Oct 30 14:53:39.200216 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:53:39.200173 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:54:09.199663 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:54:09.199628 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:54:16.169730 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.169696 6554 scope.go:122] "RemoveContainer" containerID="c1aae2d5db9089161023e0f1e98576bb74657274a17def109ac078cb7af80a8a" Oct 30 14:54:16.177823 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.177798 6554 scope.go:122] "RemoveContainer" containerID="6dde33c5de4f18713c381b4d5d0a5ecc767d771aee7b723fab08b68d69b8c998" Oct 30 14:54:16.185662 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.185635 6554 scope.go:122] "RemoveContainer" containerID="69fa3f707c2c52ddca4de1c012377a567ad42ba2eb1bf5c3345a34d41538e3f0" Oct 30 14:54:16.192408 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.192391 6554 scope.go:122] "RemoveContainer" containerID="5c997bb1c8f930ff2dc91bcb785a7a1dc18c9276f49fdf17dd924769a70b359b" Oct 30 14:54:16.198794 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.198774 6554 scope.go:122] "RemoveContainer" containerID="4fdaf243b2ba0e72e8e2210480f797faf17330681e8c0635db7a732609af7c9c" Oct 30 14:54:16.205299 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.205282 6554 scope.go:122] "RemoveContainer" containerID="5aa58ce3fee9d2dfbcbd8664ffd0ec3e1fac15d84b1669fcaf3880dd610e84ee" Oct 30 14:54:16.211872 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:16.211856 6554 scope.go:122] "RemoveContainer" containerID="3136254ef0afc29d67ec4b720c557c528cd9985a0ba39473588d90fdfd96533d" Oct 30 14:54:33.721785 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.721584 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84"] Oct 30 14:54:33.725193 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.725169 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:33.761706 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.761681 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84"] Oct 30 14:54:33.770855 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.770837 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 14:54:33.770952 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.770837 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 14:54:33.774884 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.774864 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5mb\" (UniqueName: \"kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb\") pod \"limitador-operator-controller-manager-856dd584c5-mhn84\" (UID: \"999249a3-bbb0-46cf-a62e-667d6078fa84\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:33.800342 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.800320 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x8kt5\"" Oct 30 14:54:33.876153 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.876134 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5mb\" (UniqueName: \"kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb\") pod \"limitador-operator-controller-manager-856dd584c5-mhn84\" (UID: \"999249a3-bbb0-46cf-a62e-667d6078fa84\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:33.900635 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:33.900615 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5mb\" (UniqueName: \"kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb\") pod \"limitador-operator-controller-manager-856dd584c5-mhn84\" (UID: \"999249a3-bbb0-46cf-a62e-667d6078fa84\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:34.037865 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:34.037815 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:34.338658 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:34.338584 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84"] Oct 30 14:54:34.341690 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:54:34.341645 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999249a3_bbb0_46cf_a62e_667d6078fa84.slice/crio-27077dbbb28e768e5dce2c55bc55058eca022e9b7c20dcb02b22c827bfff0c7f WatchSource:0}: Error finding container 27077dbbb28e768e5dce2c55bc55058eca022e9b7c20dcb02b22c827bfff0c7f: Status 404 returned error can't find the container with id 27077dbbb28e768e5dce2c55bc55058eca022e9b7c20dcb02b22c827bfff0c7f Oct 30 14:54:34.783063 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:34.783029 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" event={"ID":"999249a3-bbb0-46cf-a62e-667d6078fa84","Type":"ContainerStarted","Data":"27077dbbb28e768e5dce2c55bc55058eca022e9b7c20dcb02b22c827bfff0c7f"} Oct 30 14:54:37.388074 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.388047 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5"] Oct 30 14:54:37.391373 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.391339 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.411926 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.411902 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5"] Oct 30 14:54:37.416849 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.416819 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 14:54:37.424615 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.424596 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 14:54:37.424615 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.424596 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xl9hw\"" Oct 30 14:54:37.503216 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.503180 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.503313 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.503243 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.503368 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.503323 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfm9d\" (UniqueName: \"kubernetes.io/projected/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-kube-api-access-mfm9d\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.603697 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.603668 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.603789 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.603718 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.603789 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.603756 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfm9d\" (UniqueName: \"kubernetes.io/projected/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-kube-api-access-mfm9d\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.604447 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.604421 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.605860 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.605842 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.628205 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.628182 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfm9d\" (UniqueName: \"kubernetes.io/projected/7ff199bb-f4ae-408f-b5c1-3e172f42a81e-kube-api-access-mfm9d\") pod \"kuadrant-console-plugin-66c4fb575d-x7gb5\" (UID: \"7ff199bb-f4ae-408f-b5c1-3e172f42a81e\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.702680 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.702621 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" Oct 30 14:54:37.797397 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.797337 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" event={"ID":"999249a3-bbb0-46cf-a62e-667d6078fa84","Type":"ContainerStarted","Data":"358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346"} Oct 30 14:54:37.797525 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.797443 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:37.842216 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:37.842163 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" podStartSLOduration=2.163796561 podStartE2EDuration="4.842150022s" totalImagesPullingTime="2.678353461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:37.841615811 +0000 UTC m=+381.800657216" watchObservedRunningTime="2026-10-30 14:54:37.842150022 +0000 UTC m=+381.801191427" Oct 30 14:54:38.000226 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:38.000167 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5"] Oct 30 14:54:38.002799 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:54:38.002769 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ff199bb_f4ae_408f_b5c1_3e172f42a81e.slice/crio-1c7671989a4e06569317d8bf1f651d2c4d0461e526232921b56627bb3c803988 WatchSource:0}: Error finding container 1c7671989a4e06569317d8bf1f651d2c4d0461e526232921b56627bb3c803988: Status 404 returned error can't find the container with id 1c7671989a4e06569317d8bf1f651d2c4d0461e526232921b56627bb3c803988 Oct 30 14:54:38.805378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:38.805327 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" event={"ID":"7ff199bb-f4ae-408f-b5c1-3e172f42a81e","Type":"ContainerStarted","Data":"1c7671989a4e06569317d8bf1f651d2c4d0461e526232921b56627bb3c803988"} Oct 30 14:54:39.209088 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:54:39.209045 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:54:42.827151 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:42.827125 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" event={"ID":"7ff199bb-f4ae-408f-b5c1-3e172f42a81e","Type":"ContainerStarted","Data":"e7c1a027a5a5a809f6bb732bb0275cfd6c9a1fe41856a9d781f39cf64b32800a"} Oct 30 14:54:42.870084 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:42.870041 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-x7gb5" podStartSLOduration=1.236505507 podStartE2EDuration="5.870025913s" totalImagesPullingTime="4.633520406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:42.868997627 +0000 UTC m=+386.828039031" watchObservedRunningTime="2026-10-30 14:54:42.870025913 +0000 UTC m=+386.829067332" Oct 30 14:54:48.807336 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:48.807300 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:49.794474 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.794444 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84"] Oct 30 14:54:49.794693 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.794669 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" containerName="manager" containerID="cri-o://358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346" gracePeriod=2 Oct 30 14:54:49.808742 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.808718 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84"] Oct 30 14:54:49.811173 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:54:49.811145 6554 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mhn84\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:49.828226 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.828199 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv"] Oct 30 14:54:49.828562 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.828547 6554 cpu_manager.go:414] "containerMap: removing container" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" containerName="manager" Oct 30 14:54:49.828636 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.828563 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" containerName="manager" Oct 30 14:54:49.828694 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.828644 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" containerName="manager" Oct 30 14:54:49.831661 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.831642 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:54:49.846755 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:54:49.846725 6554 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mhn84\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:49.858946 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.858924 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv"] Oct 30 14:54:49.913176 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:49.913149 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2xqr\" (UniqueName: \"kubernetes.io/projected/6875de95-ef0d-4239-a549-74086c0d5b0f-kube-api-access-x2xqr\") pod \"limitador-operator-controller-manager-856dd584c5-8mhhv\" (UID: \"6875de95-ef0d-4239-a549-74086c0d5b0f\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:54:50.013776 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.013747 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2xqr\" (UniqueName: \"kubernetes.io/projected/6875de95-ef0d-4239-a549-74086c0d5b0f-kube-api-access-x2xqr\") pod \"limitador-operator-controller-manager-856dd584c5-8mhhv\" (UID: \"6875de95-ef0d-4239-a549-74086c0d5b0f\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:54:50.032818 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.032793 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:50.040899 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.040877 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2xqr\" (UniqueName: \"kubernetes.io/projected/6875de95-ef0d-4239-a549-74086c0d5b0f-kube-api-access-x2xqr\") pod \"limitador-operator-controller-manager-856dd584c5-8mhhv\" (UID: \"6875de95-ef0d-4239-a549-74086c0d5b0f\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:54:50.058533 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:54:50.058480 6554 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-mhn84\" is forbidden: User \"system:node:ip-10-0-139-204.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-204.ec2.internal' and this object" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:50.115123 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.115095 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb\" (UniqueName: \"kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb\") pod \"999249a3-bbb0-46cf-a62e-667d6078fa84\" (UID: \"999249a3-bbb0-46cf-a62e-667d6078fa84\") " Oct 30 14:54:50.117041 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.117016 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb" pod "999249a3-bbb0-46cf-a62e-667d6078fa84" (UID: "999249a3-bbb0-46cf-a62e-667d6078fa84"). InnerVolumeSpecName "kube-api-access-hc5mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:54:50.151786 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.151755 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999249a3-bbb0-46cf-a62e-667d6078fa84" path="/var/lib/kubelet/pods/999249a3-bbb0-46cf-a62e-667d6078fa84/volumes" Oct 30 14:54:50.180481 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.180463 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:54:50.216791 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.216758 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc5mb\" (UniqueName: \"kubernetes.io/projected/999249a3-bbb0-46cf-a62e-667d6078fa84-kube-api-access-hc5mb\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:54:50.479514 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.479484 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv"] Oct 30 14:54:50.482452 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:54:50.482425 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6875de95_ef0d_4239_a549_74086c0d5b0f.slice/crio-624277c7dc42b48959aafcbaa39422dfa444e3c0d48027cc508e9b76a5f42e36 WatchSource:0}: Error finding container 624277c7dc42b48959aafcbaa39422dfa444e3c0d48027cc508e9b76a5f42e36: Status 404 returned error can't find the container with id 624277c7dc42b48959aafcbaa39422dfa444e3c0d48027cc508e9b76a5f42e36 Oct 30 14:54:50.861475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.861398 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" event={"ID":"6875de95-ef0d-4239-a549-74086c0d5b0f","Type":"ContainerStarted","Data":"567e55717d52aadb0435d139c55da012b19a3b7697e20a7551f1dd67a78ef656"} Oct 30 14:54:50.861475 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.861449 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" event={"ID":"6875de95-ef0d-4239-a549-74086c0d5b0f","Type":"ContainerStarted","Data":"624277c7dc42b48959aafcbaa39422dfa444e3c0d48027cc508e9b76a5f42e36"} Oct 30 14:54:50.861925 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.861477 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:54:50.862658 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.862639 6554 generic.go:356] "Generic (PLEG): container finished" podID="999249a3-bbb0-46cf-a62e-667d6078fa84" containerID="358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346" exitCode=0 Oct 30 14:54:50.862721 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.862670 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-mhn84" Oct 30 14:54:50.862721 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.862680 6554 scope.go:122] "RemoveContainer" containerID="358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346" Oct 30 14:54:50.870117 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.870099 6554 scope.go:122] "RemoveContainer" containerID="358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346" Oct 30 14:54:50.870404 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:54:50.870384 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346\": container with ID starting with 358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346 not found: ID does not exist" containerID="358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346" Oct 30 14:54:50.870501 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.870409 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346"} err="failed to get container status \"358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346\": rpc error: code = NotFound desc = could not find container \"358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346\": container with ID starting with 358d3fbdd70a777b20668768f8e2c76dcab810f6bf85d67b3918eea0e73b8346 not found: ID does not exist" Oct 30 14:54:50.957629 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:54:50.957058 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" podStartSLOduration=1.957041961 podStartE2EDuration="1.957041961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:54:50.956381817 +0000 UTC m=+394.915423214" watchObservedRunningTime="2026-10-30 14:54:50.957041961 +0000 UTC m=+394.916083364" Oct 30 14:55:01.871258 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:01.871227 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-8mhhv" Oct 30 14:55:09.200320 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:55:09.200280 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:55:32.997986 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:32.997947 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rkqjr"] Oct 30 14:55:33.001454 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.001435 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:33.019254 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.019226 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rkqjr"] Oct 30 14:55:33.031748 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.031723 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jz9gp\"" Oct 30 14:55:33.043759 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.043739 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqgrt\" (UniqueName: \"kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt\") pod \"authorino-57f7f8659f-rkqjr\" (UID: \"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c\") " pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:33.144232 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.144192 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqgrt\" (UniqueName: \"kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt\") pod \"authorino-57f7f8659f-rkqjr\" (UID: \"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c\") " pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:33.168518 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.168494 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqgrt\" (UniqueName: \"kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt\") pod \"authorino-57f7f8659f-rkqjr\" (UID: \"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c\") " pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:33.319010 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.318955 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:33.614515 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.614445 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rkqjr"] Oct 30 14:55:33.617807 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:55:33.617773 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod71f9c556_8ba4_41c2_bd03_fc88d1cfe74c.slice/crio-3cc05b4b725d9254725cfaf5753e5d72e65f2d0cff545337b33e316d2b97f9d6 WatchSource:0}: Error finding container 3cc05b4b725d9254725cfaf5753e5d72e65f2d0cff545337b33e316d2b97f9d6: Status 404 returned error can't find the container with id 3cc05b4b725d9254725cfaf5753e5d72e65f2d0cff545337b33e316d2b97f9d6 Oct 30 14:55:33.751981 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.751957 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mmldf"] Oct 30 14:55:33.756431 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.756412 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:55:33.775815 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.775789 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mmldf"] Oct 30 14:55:33.850051 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.850028 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzpkn\" (UniqueName: \"kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn\") pod \"authorino-56c6c9ddd5-mmldf\" (UID: \"9933d314-cf50-4a35-92ca-aa4a7025341d\") " pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:55:33.950704 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.950680 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzpkn\" (UniqueName: \"kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn\") pod \"authorino-56c6c9ddd5-mmldf\" (UID: \"9933d314-cf50-4a35-92ca-aa4a7025341d\") " pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:55:33.978429 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:33.978405 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzpkn\" (UniqueName: \"kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn\") pod \"authorino-56c6c9ddd5-mmldf\" (UID: \"9933d314-cf50-4a35-92ca-aa4a7025341d\") " pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:55:34.049773 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:34.049746 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" event={"ID":"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c","Type":"ContainerStarted","Data":"3cc05b4b725d9254725cfaf5753e5d72e65f2d0cff545337b33e316d2b97f9d6"} Oct 30 14:55:34.067110 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:34.067086 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:55:34.358558 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:34.358206 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mmldf"] Oct 30 14:55:34.361007 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:55:34.360979 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9933d314_cf50_4a35_92ca_aa4a7025341d.slice/crio-febe25d9b9fb4bf0b459c0a9e88da0b5f261196c5c2f588caf0205b4bd99f704 WatchSource:0}: Error finding container febe25d9b9fb4bf0b459c0a9e88da0b5f261196c5c2f588caf0205b4bd99f704: Status 404 returned error can't find the container with id febe25d9b9fb4bf0b459c0a9e88da0b5f261196c5c2f588caf0205b4bd99f704 Oct 30 14:55:35.055698 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:35.055664 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" event={"ID":"9933d314-cf50-4a35-92ca-aa4a7025341d","Type":"ContainerStarted","Data":"febe25d9b9fb4bf0b459c0a9e88da0b5f261196c5c2f588caf0205b4bd99f704"} Oct 30 14:55:37.065406 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:37.065361 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" event={"ID":"9933d314-cf50-4a35-92ca-aa4a7025341d","Type":"ContainerStarted","Data":"fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73"} Oct 30 14:55:37.107154 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:37.107076 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" podStartSLOduration=2.405566077 podStartE2EDuration="4.1070609s" totalImagesPullingTime="1.701494823s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:37.106863275 +0000 UTC m=+441.065904721" watchObservedRunningTime="2026-10-30 14:55:37.1070609 +0000 UTC m=+441.066102305" Oct 30 14:55:37.134196 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:37.134167 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rkqjr"] Oct 30 14:55:39.200186 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:55:39.200147 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:55:42.088973 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.088897 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" event={"ID":"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c","Type":"ContainerStarted","Data":"24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554"} Oct 30 14:55:42.089427 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.088960 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" podUID="71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" containerName="authorino" containerID="cri-o://24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554" gracePeriod=30 Oct 30 14:55:42.129999 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.129944 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" podStartSLOduration=1.9131253799999999 podStartE2EDuration="10.129932797s" totalImagesPullingTime="8.216807417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:55:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:55:42.129866517 +0000 UTC m=+446.088907922" watchObservedRunningTime="2026-10-30 14:55:42.129932797 +0000 UTC m=+446.088974201" Oct 30 14:55:42.320677 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.320655 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:42.409410 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.409391 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt\" (UniqueName: \"kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt\") pod \"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c\" (UID: \"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c\") " Oct 30 14:55:42.411359 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.411322 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt" pod "71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" (UID: "71f9c556-8ba4-41c2-bd03-fc88d1cfe74c"). InnerVolumeSpecName "kube-api-access-mqgrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:55:42.510676 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:42.510640 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqgrt\" (UniqueName: \"kubernetes.io/projected/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c-kube-api-access-mqgrt\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:55:43.095045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.095013 6554 generic.go:356] "Generic (PLEG): container finished" podID="71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" containerID="24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554" exitCode=0 Oct 30 14:55:43.095447 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.095062 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" Oct 30 14:55:43.095447 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.095108 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" event={"ID":"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c","Type":"ContainerDied","Data":"24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554"} Oct 30 14:55:43.095447 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.095155 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-rkqjr" event={"ID":"71f9c556-8ba4-41c2-bd03-fc88d1cfe74c","Type":"ContainerDied","Data":"3cc05b4b725d9254725cfaf5753e5d72e65f2d0cff545337b33e316d2b97f9d6"} Oct 30 14:55:43.095447 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.095173 6554 scope.go:122] "RemoveContainer" containerID="24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554" Oct 30 14:55:43.104432 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.104416 6554 scope.go:122] "RemoveContainer" containerID="24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554" Oct 30 14:55:43.104897 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:55:43.104876 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554\": container with ID starting with 24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554 not found: ID does not exist" containerID="24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554" Oct 30 14:55:43.105038 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.105016 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554"} err="failed to get container status \"24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554\": rpc error: code = NotFound desc = could not find container \"24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554\": container with ID starting with 24c7fc4d6520582beef71f3732e8c2b0ff44856ca838514d453482bc5f85c554 not found: ID does not exist" Oct 30 14:55:43.149058 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.149036 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rkqjr"] Oct 30 14:55:43.162463 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:43.162437 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-rkqjr"] Oct 30 14:55:44.152089 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:55:44.152060 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" path="/var/lib/kubelet/pods/71f9c556-8ba4-41c2-bd03-fc88d1cfe74c/volumes" Oct 30 14:56:02.981710 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:02.981617 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ghrz8"] Oct 30 14:56:02.982198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:02.982104 6554 cpu_manager.go:414] "containerMap: removing container" podUID="71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" containerName="authorino" Oct 30 14:56:02.982198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:02.982125 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" containerName="authorino" Oct 30 14:56:02.982327 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:02.982224 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f9c556-8ba4-41c2-bd03-fc88d1cfe74c" containerName="authorino" Oct 30 14:56:02.985151 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:02.985131 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:03.001962 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.001939 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ghrz8"] Oct 30 14:56:03.066732 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.066706 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6686l\" (UniqueName: \"kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l\") pod \"authorino-6f6944bcdd-ghrz8\" (UID: \"28a26926-7fe1-424f-b7fb-d93ebbd82bb0\") " pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:03.167641 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.167610 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6686l\" (UniqueName: \"kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l\") pod \"authorino-6f6944bcdd-ghrz8\" (UID: \"28a26926-7fe1-424f-b7fb-d93ebbd82bb0\") " pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:03.195953 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.195930 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6686l\" (UniqueName: \"kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l\") pod \"authorino-6f6944bcdd-ghrz8\" (UID: \"28a26926-7fe1-424f-b7fb-d93ebbd82bb0\") " pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:03.196041 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.196025 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ghrz8"] Oct 30 14:56:03.196184 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.196173 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:03.228943 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.228920 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hkz7m"] Oct 30 14:56:03.237234 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.237172 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.253867 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.253839 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hkz7m"] Oct 30 14:56:03.254099 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.254080 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 14:56:03.328410 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.328383 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hkz7m"] Oct 30 14:56:03.328589 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:56:03.328571 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rvvpb tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" podUID="7ebd685d-183a-4e89-9f3b-8293c8d34853" Oct 30 14:56:03.369872 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.369837 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert\") pod \"authorino-6889cbd8c4-hkz7m\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.369973 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.369906 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvvpb\" (UniqueName: \"kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb\") pod \"authorino-6889cbd8c4-hkz7m\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.389299 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.389273 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77cffb447b-5qfb5"] Oct 30 14:56:03.392657 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.392642 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.465244 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.465220 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77cffb447b-5qfb5"] Oct 30 14:56:03.470708 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.470683 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert\") pod \"authorino-6889cbd8c4-hkz7m\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.470792 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.470733 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert\") pod \"authorino-77cffb447b-5qfb5\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.470833 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.470804 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvvpb\" (UniqueName: \"kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb\") pod \"authorino-6889cbd8c4-hkz7m\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.470897 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.470875 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc\") pod \"authorino-77cffb447b-5qfb5\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.473106 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.473079 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert\") pod \"authorino-6889cbd8c4-hkz7m\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.495895 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.495838 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ghrz8"] Oct 30 14:56:03.498426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.498408 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvvpb\" (UniqueName: \"kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb\") pod \"authorino-6889cbd8c4-hkz7m\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:03.500266 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:56:03.500245 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28a26926_7fe1_424f_b7fb_d93ebbd82bb0.slice/crio-7e5e15fb76d0bbe5d728f216b7cddc82507dd1f75ee110c4877aa3e3b114e96d WatchSource:0}: Error finding container 7e5e15fb76d0bbe5d728f216b7cddc82507dd1f75ee110c4877aa3e3b114e96d: Status 404 returned error can't find the container with id 7e5e15fb76d0bbe5d728f216b7cddc82507dd1f75ee110c4877aa3e3b114e96d Oct 30 14:56:03.572164 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.572144 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert\") pod \"authorino-77cffb447b-5qfb5\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.572253 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.572190 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc\") pod \"authorino-77cffb447b-5qfb5\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.574328 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.574307 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert\") pod \"authorino-77cffb447b-5qfb5\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.597296 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.597276 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc\") pod \"authorino-77cffb447b-5qfb5\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.703270 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.703250 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:56:03.997299 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:03.997277 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77cffb447b-5qfb5"] Oct 30 14:56:04.004455 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:56:04.004429 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod464ec6bb_108c_41fe_accd_83589d9ec50e.slice/crio-2bf8f055d8f88f00818848183fd018d4a3931c90dc17d0f6d5eabc1ac3664134 WatchSource:0}: Error finding container 2bf8f055d8f88f00818848183fd018d4a3931c90dc17d0f6d5eabc1ac3664134: Status 404 returned error can't find the container with id 2bf8f055d8f88f00818848183fd018d4a3931c90dc17d0f6d5eabc1ac3664134 Oct 30 14:56:04.188522 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.188484 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cffb447b-5qfb5" event={"ID":"464ec6bb-108c-41fe-accd-83589d9ec50e","Type":"ContainerStarted","Data":"2bf8f055d8f88f00818848183fd018d4a3931c90dc17d0f6d5eabc1ac3664134"} Oct 30 14:56:04.189925 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.189901 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:04.190039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.189937 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" event={"ID":"28a26926-7fe1-424f-b7fb-d93ebbd82bb0","Type":"ContainerStarted","Data":"10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee"} Oct 30 14:56:04.190039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.189959 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" podUID="28a26926-7fe1-424f-b7fb-d93ebbd82bb0" containerName="authorino" containerID="cri-o://10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee" gracePeriod=30 Oct 30 14:56:04.190039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.189970 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" event={"ID":"28a26926-7fe1-424f-b7fb-d93ebbd82bb0","Type":"ContainerStarted","Data":"7e5e15fb76d0bbe5d728f216b7cddc82507dd1f75ee110c4877aa3e3b114e96d"} Oct 30 14:56:04.196807 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.196787 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:04.266856 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.266783 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" podStartSLOduration=1.78963456 podStartE2EDuration="2.266770509s" totalImagesPullingTime="477.135949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:04.266398 +0000 UTC m=+468.225439406" watchObservedRunningTime="2026-10-30 14:56:04.266770509 +0000 UTC m=+468.225811913" Oct 30 14:56:04.278597 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.278578 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert\" (UniqueName: \"kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert\") pod \"7ebd685d-183a-4e89-9f3b-8293c8d34853\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " Oct 30 14:56:04.278687 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.278612 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb\" (UniqueName: \"kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb\") pod \"7ebd685d-183a-4e89-9f3b-8293c8d34853\" (UID: \"7ebd685d-183a-4e89-9f3b-8293c8d34853\") " Oct 30 14:56:04.280647 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.280623 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb" pod "7ebd685d-183a-4e89-9f3b-8293c8d34853" (UID: "7ebd685d-183a-4e89-9f3b-8293c8d34853"). InnerVolumeSpecName "kube-api-access-rvvpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:56:04.280746 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.280663 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert" pod "7ebd685d-183a-4e89-9f3b-8293c8d34853" (UID: "7ebd685d-183a-4e89-9f3b-8293c8d34853"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:56:04.379869 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.379846 6554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7ebd685d-183a-4e89-9f3b-8293c8d34853-tls-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:56:04.379869 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.379871 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvvpb\" (UniqueName: \"kubernetes.io/projected/7ebd685d-183a-4e89-9f3b-8293c8d34853-kube-api-access-rvvpb\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:56:04.429175 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.429154 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:04.581653 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.581583 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l\" (UniqueName: \"kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l\") pod \"28a26926-7fe1-424f-b7fb-d93ebbd82bb0\" (UID: \"28a26926-7fe1-424f-b7fb-d93ebbd82bb0\") " Oct 30 14:56:04.583363 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.583331 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l" pod "28a26926-7fe1-424f-b7fb-d93ebbd82bb0" (UID: "28a26926-7fe1-424f-b7fb-d93ebbd82bb0"). InnerVolumeSpecName "kube-api-access-6686l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:56:04.682718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:04.682690 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6686l\" (UniqueName: \"kubernetes.io/projected/28a26926-7fe1-424f-b7fb-d93ebbd82bb0-kube-api-access-6686l\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:56:05.195225 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.195193 6554 generic.go:356] "Generic (PLEG): container finished" podID="28a26926-7fe1-424f-b7fb-d93ebbd82bb0" containerID="10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee" exitCode=0 Oct 30 14:56:05.195718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.195227 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" Oct 30 14:56:05.195718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.195281 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" event={"ID":"28a26926-7fe1-424f-b7fb-d93ebbd82bb0","Type":"ContainerDied","Data":"10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee"} Oct 30 14:56:05.195718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.195317 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ghrz8" event={"ID":"28a26926-7fe1-424f-b7fb-d93ebbd82bb0","Type":"ContainerDied","Data":"7e5e15fb76d0bbe5d728f216b7cddc82507dd1f75ee110c4877aa3e3b114e96d"} Oct 30 14:56:05.195718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.195335 6554 scope.go:122] "RemoveContainer" containerID="10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee" Oct 30 14:56:05.196990 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.196960 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hkz7m" Oct 30 14:56:05.197103 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.196994 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cffb447b-5qfb5" event={"ID":"464ec6bb-108c-41fe-accd-83589d9ec50e","Type":"ContainerStarted","Data":"23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610"} Oct 30 14:56:05.212156 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.212140 6554 scope.go:122] "RemoveContainer" containerID="10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee" Oct 30 14:56:05.213048 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:56:05.213027 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee\": container with ID starting with 10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee not found: ID does not exist" containerID="10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee" Oct 30 14:56:05.213133 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.213057 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee"} err="failed to get container status \"10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee\": rpc error: code = NotFound desc = could not find container \"10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee\": container with ID starting with 10407daa52577dac33b15a4c7dd2b2e0877ea861d3f0ef0ffd860f40156bfbee not found: ID does not exist" Oct 30 14:56:05.255090 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.255046 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-77cffb447b-5qfb5" podStartSLOduration=1.817962772 podStartE2EDuration="2.25503466s" totalImagesPullingTime="437.071888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:56:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:56:05.254393805 +0000 UTC m=+469.213435208" watchObservedRunningTime="2026-10-30 14:56:05.25503466 +0000 UTC m=+469.214076064" Oct 30 14:56:05.282717 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.282697 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mmldf"] Oct 30 14:56:05.282855 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.282838 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" podUID="9933d314-cf50-4a35-92ca-aa4a7025341d" containerName="authorino" containerID="cri-o://fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73" gracePeriod=30 Oct 30 14:56:05.298533 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.298492 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ghrz8"] Oct 30 14:56:05.298854 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.298824 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ghrz8"] Oct 30 14:56:05.326342 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:56:05.326315 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9933d314_cf50_4a35_92ca_aa4a7025341d.slice/crio-conmon-fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73.scope\": RecentStats: unable to find data in memory cache]" Oct 30 14:56:05.360683 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.360658 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hkz7m"] Oct 30 14:56:05.364419 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.364396 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hkz7m"] Oct 30 14:56:05.522870 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.522849 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:56:05.691572 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.691543 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn\" (UniqueName: \"kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn\") pod \"9933d314-cf50-4a35-92ca-aa4a7025341d\" (UID: \"9933d314-cf50-4a35-92ca-aa4a7025341d\") " Oct 30 14:56:05.693512 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.693486 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn" pod "9933d314-cf50-4a35-92ca-aa4a7025341d" (UID: "9933d314-cf50-4a35-92ca-aa4a7025341d"). InnerVolumeSpecName "kube-api-access-bzpkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:56:05.792528 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:05.792492 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzpkn\" (UniqueName: \"kubernetes.io/projected/9933d314-cf50-4a35-92ca-aa4a7025341d-kube-api-access-bzpkn\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:56:06.151627 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.151599 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a26926-7fe1-424f-b7fb-d93ebbd82bb0" path="/var/lib/kubelet/pods/28a26926-7fe1-424f-b7fb-d93ebbd82bb0/volumes" Oct 30 14:56:06.151931 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.151918 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ebd685d-183a-4e89-9f3b-8293c8d34853" path="/var/lib/kubelet/pods/7ebd685d-183a-4e89-9f3b-8293c8d34853/volumes" Oct 30 14:56:06.203124 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.203095 6554 generic.go:356] "Generic (PLEG): container finished" podID="9933d314-cf50-4a35-92ca-aa4a7025341d" containerID="fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73" exitCode=0 Oct 30 14:56:06.203506 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.203143 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" Oct 30 14:56:06.203506 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.203170 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" event={"ID":"9933d314-cf50-4a35-92ca-aa4a7025341d","Type":"ContainerDied","Data":"fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73"} Oct 30 14:56:06.203506 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.203220 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mmldf" event={"ID":"9933d314-cf50-4a35-92ca-aa4a7025341d","Type":"ContainerDied","Data":"febe25d9b9fb4bf0b459c0a9e88da0b5f261196c5c2f588caf0205b4bd99f704"} Oct 30 14:56:06.203506 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.203249 6554 scope.go:122] "RemoveContainer" containerID="fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73" Oct 30 14:56:06.211109 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.211090 6554 scope.go:122] "RemoveContainer" containerID="fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73" Oct 30 14:56:06.211411 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:56:06.211364 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73\": container with ID starting with fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73 not found: ID does not exist" containerID="fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73" Oct 30 14:56:06.211411 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.211399 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73"} err="failed to get container status \"fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73\": rpc error: code = NotFound desc = could not find container \"fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73\": container with ID starting with fa2f054df805ee70805d342587b6edd748ef11edc70f7f39b3cbbbd474670a73 not found: ID does not exist" Oct 30 14:56:06.262541 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.262516 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mmldf"] Oct 30 14:56:06.273514 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:06.270037 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mmldf"] Oct 30 14:56:08.152796 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:56:08.152754 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9933d314-cf50-4a35-92ca-aa4a7025341d" path="/var/lib/kubelet/pods/9933d314-cf50-4a35-92ca-aa4a7025341d/volumes" Oct 30 14:56:09.202433 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:56:09.202392 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:56:39.201387 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:56:39.201338 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:57:09.200499 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:57:09.200457 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:57:39.200242 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:57:39.200202 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:57:59.431550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.431513 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm"] Oct 30 14:57:59.432075 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.432055 6554 cpu_manager.go:414] "containerMap: removing container" podUID="28a26926-7fe1-424f-b7fb-d93ebbd82bb0" containerName="authorino" Oct 30 14:57:59.432118 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.432079 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a26926-7fe1-424f-b7fb-d93ebbd82bb0" containerName="authorino" Oct 30 14:57:59.432118 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.432113 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9933d314-cf50-4a35-92ca-aa4a7025341d" containerName="authorino" Oct 30 14:57:59.432183 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.432121 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9933d314-cf50-4a35-92ca-aa4a7025341d" containerName="authorino" Oct 30 14:57:59.432214 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.432205 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a26926-7fe1-424f-b7fb-d93ebbd82bb0" containerName="authorino" Oct 30 14:57:59.432246 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.432219 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9933d314-cf50-4a35-92ca-aa4a7025341d" containerName="authorino" Oct 30 14:57:59.435596 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.435576 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.452215 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.452191 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9r8cs\"" Oct 30 14:57:59.452313 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.452223 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 14:57:59.467008 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.466988 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm"] Oct 30 14:57:59.495028 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.495004 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 14:57:59.495137 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.495071 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 14:57:59.632797 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.632773 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgf8n\" (UniqueName: \"kubernetes.io/projected/caa663cd-eb7c-4b24-8601-4a9256fd097c-kube-api-access-rgf8n\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.632925 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.632809 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.632925 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.632833 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.633001 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.632946 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.633001 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.632977 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.633078 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.633016 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa663cd-eb7c-4b24-8601-4a9256fd097c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.733567 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.733510 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa663cd-eb7c-4b24-8601-4a9256fd097c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.733567 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.733557 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgf8n\" (UniqueName: \"kubernetes.io/projected/caa663cd-eb7c-4b24-8601-4a9256fd097c-kube-api-access-rgf8n\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.733750 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.733725 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.733808 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.733783 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.733854 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.733836 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.733900 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.733869 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.734143 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.734119 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.734270 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.734147 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.734270 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.734202 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.735927 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.735906 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa663cd-eb7c-4b24-8601-4a9256fd097c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.735927 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.735915 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa663cd-eb7c-4b24-8601-4a9256fd097c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:57:59.758843 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:57:59.758819 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgf8n\" (UniqueName: \"kubernetes.io/projected/caa663cd-eb7c-4b24-8601-4a9256fd097c-kube-api-access-rgf8n\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-8rxfm\" (UID: \"caa663cd-eb7c-4b24-8601-4a9256fd097c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:58:00.050015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:00.049959 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:58:00.346893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:00.346760 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm"] Oct 30 14:58:00.349009 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:00.348986 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa663cd_eb7c_4b24_8601_4a9256fd097c.slice/crio-eb2eca55db49b96091504ec4d6dbd81e50119840001ad0b4ef7509adf1647c66 WatchSource:0}: Error finding container eb2eca55db49b96091504ec4d6dbd81e50119840001ad0b4ef7509adf1647c66: Status 404 returned error can't find the container with id eb2eca55db49b96091504ec4d6dbd81e50119840001ad0b4ef7509adf1647c66 Oct 30 14:58:00.350240 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:00.350223 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 14:58:00.671202 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:00.671160 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" event={"ID":"caa663cd-eb7c-4b24-8601-4a9256fd097c","Type":"ContainerStarted","Data":"eb2eca55db49b96091504ec4d6dbd81e50119840001ad0b4ef7509adf1647c66"} Oct 30 14:58:01.679566 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:01.679530 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" event={"ID":"caa663cd-eb7c-4b24-8601-4a9256fd097c","Type":"ContainerStarted","Data":"e77f5baaa3374582cf4490149d67e63cf1d4c611ac7ce50adc68554c204d7dc0"} Oct 30 14:58:01.727677 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:01.727624 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" podStartSLOduration=1.506729391 podStartE2EDuration="2.727608367s" totalImagesPullingTime="1.220878976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:01.7266707 +0000 UTC m=+585.685712106" watchObservedRunningTime="2026-10-30 14:58:01.727608367 +0000 UTC m=+585.686649796" Oct 30 14:58:02.050512 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:02.050481 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:58:03.107941 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.107908 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq"] Oct 30 14:58:03.111381 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.111344 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.129570 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.129546 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 14:58:03.144584 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.144560 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq"] Oct 30 14:58:03.163330 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.163305 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.163463 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.163380 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.163463 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.163446 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5h66\" (UniqueName: \"kubernetes.io/projected/ae429169-b691-4914-8603-c992a490012c-kube-api-access-h5h66\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.163554 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.163473 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.163554 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.163506 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.163554 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.163525 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae429169-b691-4914-8603-c992a490012c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.264582 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.264549 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.264770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.264600 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.264770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.264622 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5h66\" (UniqueName: \"kubernetes.io/projected/ae429169-b691-4914-8603-c992a490012c-kube-api-access-h5h66\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.264770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.264645 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.264770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.264684 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.264770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.264702 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae429169-b691-4914-8603-c992a490012c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.265090 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.265026 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.265147 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.265084 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.265191 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.265168 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.267066 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.267042 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae429169-b691-4914-8603-c992a490012c-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.267235 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.267218 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae429169-b691-4914-8603-c992a490012c-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.290974 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.290942 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5h66\" (UniqueName: \"kubernetes.io/projected/ae429169-b691-4914-8603-c992a490012c-kube-api-access-h5h66\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-pspnq\" (UID: \"ae429169-b691-4914-8603-c992a490012c\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.425121 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.424958 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:03.733074 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:03.733044 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq"] Oct 30 14:58:03.735280 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:03.735255 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae429169_b691_4914_8603_c992a490012c.slice/crio-dc21a7613a83f44628137c8ed901a0b2105033a8f21fda4f05f8d268b82fac16 WatchSource:0}: Error finding container dc21a7613a83f44628137c8ed901a0b2105033a8f21fda4f05f8d268b82fac16: Status 404 returned error can't find the container with id dc21a7613a83f44628137c8ed901a0b2105033a8f21fda4f05f8d268b82fac16 Oct 30 14:58:04.698076 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:04.698040 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" event={"ID":"ae429169-b691-4914-8603-c992a490012c","Type":"ContainerStarted","Data":"031c5b571e75487f7342367e229bb000ef0ab6d9e1e5e8ec73382a4e91f7272e"} Oct 30 14:58:04.698478 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:04.698090 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" event={"ID":"ae429169-b691-4914-8603-c992a490012c","Type":"ContainerStarted","Data":"dc21a7613a83f44628137c8ed901a0b2105033a8f21fda4f05f8d268b82fac16"} Oct 30 14:58:04.744219 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:04.744168 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" podStartSLOduration=1.516469777 podStartE2EDuration="1.744156213s" totalImagesPullingTime="227.686436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:04.743627338 +0000 UTC m=+588.702668743" watchObservedRunningTime="2026-10-30 14:58:04.744156213 +0000 UTC m=+588.703197618" Oct 30 14:58:05.425313 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:05.425277 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:07.644891 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.644849 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn"] Oct 30 14:58:07.648692 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.648661 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.678785 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.678756 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 14:58:07.680177 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.680146 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn"] Oct 30 14:58:07.703943 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.703916 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbhs\" (UniqueName: \"kubernetes.io/projected/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-kube-api-access-fnbhs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.704107 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.703950 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.704107 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.703977 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.704211 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.704132 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.704211 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.704165 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.704305 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.704260 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805168 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805144 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbhs\" (UniqueName: \"kubernetes.io/projected/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-kube-api-access-fnbhs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805261 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805175 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805261 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805197 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805345 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805320 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805431 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805373 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805431 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805417 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805566 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805546 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805716 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805697 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.805778 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.805767 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.807374 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.807331 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.807717 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.807700 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.829470 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.829451 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbhs\" (UniqueName: \"kubernetes.io/projected/2573e9d0-9c8d-4dfa-8336-0b0804094d4f-kube-api-access-fnbhs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn\" (UID: \"2573e9d0-9c8d-4dfa-8336-0b0804094d4f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:07.964224 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:07.964170 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:08.261216 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:08.261142 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn"] Oct 30 14:58:08.265571 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:08.265530 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2573e9d0_9c8d_4dfa_8336_0b0804094d4f.slice/crio-fe38e9f271b49cd8b2586ad484f9d93de690ecd29a7d5e36ad6554c99ddf3111 WatchSource:0}: Error finding container fe38e9f271b49cd8b2586ad484f9d93de690ecd29a7d5e36ad6554c99ddf3111: Status 404 returned error can't find the container with id fe38e9f271b49cd8b2586ad484f9d93de690ecd29a7d5e36ad6554c99ddf3111 Oct 30 14:58:08.716240 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:08.716187 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" event={"ID":"2573e9d0-9c8d-4dfa-8336-0b0804094d4f","Type":"ContainerStarted","Data":"9ccd39939d280f34fa508243cfb48fe3e3e1add4dd02c2949d16cd6362334316"} Oct 30 14:58:08.716736 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:08.716268 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" event={"ID":"2573e9d0-9c8d-4dfa-8336-0b0804094d4f","Type":"ContainerStarted","Data":"fe38e9f271b49cd8b2586ad484f9d93de690ecd29a7d5e36ad6554c99ddf3111"} Oct 30 14:58:08.766023 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:08.765925 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" podStartSLOduration=1.5243028280000002 podStartE2EDuration="1.765910625s" totalImagesPullingTime="241.607797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:08.764906039 +0000 UTC m=+592.723947445" watchObservedRunningTime="2026-10-30 14:58:08.765910625 +0000 UTC m=+592.724952030" Oct 30 14:58:08.965002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:08.964970 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:09.201776 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:58:09.201735 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:58:10.050881 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.050845 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:58:10.063404 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.063378 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:58:10.553072 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.553036 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh"] Oct 30 14:58:10.556990 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.556964 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.577574 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.577534 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 14:58:10.578973 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.578947 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh"] Oct 30 14:58:10.629443 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.629418 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.629558 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.629449 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.629558 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.629482 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.629700 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.629676 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72284b4e-d5b0-4317-b86f-7341f3d3364f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.629761 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.629742 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgq5\" (UniqueName: \"kubernetes.io/projected/72284b4e-d5b0-4317-b86f-7341f3d3364f-kube-api-access-tdgq5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.629814 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.629780 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.730559 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.730535 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72284b4e-d5b0-4317-b86f-7341f3d3364f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.730693 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.730576 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgq5\" (UniqueName: \"kubernetes.io/projected/72284b4e-d5b0-4317-b86f-7341f3d3364f-kube-api-access-tdgq5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.730693 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.730596 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.730693 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.730621 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.730693 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.730644 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.730693 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.730671 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.731276 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.731069 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.731276 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.731159 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.731276 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.731217 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.733031 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.733013 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72284b4e-d5b0-4317-b86f-7341f3d3364f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.733303 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.733282 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72284b4e-d5b0-4317-b86f-7341f3d3364f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.735495 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.735475 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-8rxfm" Oct 30 14:58:10.756319 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.756296 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgq5\" (UniqueName: \"kubernetes.io/projected/72284b4e-d5b0-4317-b86f-7341f3d3364f-kube-api-access-tdgq5\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh\" (UID: \"72284b4e-d5b0-4317-b86f-7341f3d3364f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:10.871612 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:10.871521 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:11.175408 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:11.175384 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh"] Oct 30 14:58:11.177934 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:11.177908 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72284b4e_d5b0_4317_b86f_7341f3d3364f.slice/crio-dfffa621c7e9cfb0339a5dbe9adc0733123e25577b40f158fec4987402b59ea4 WatchSource:0}: Error finding container dfffa621c7e9cfb0339a5dbe9adc0733123e25577b40f158fec4987402b59ea4: Status 404 returned error can't find the container with id dfffa621c7e9cfb0339a5dbe9adc0733123e25577b40f158fec4987402b59ea4 Oct 30 14:58:11.758620 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:11.758537 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" event={"ID":"72284b4e-d5b0-4317-b86f-7341f3d3364f","Type":"ContainerStarted","Data":"334739623f334b8f0b249ad6a705bea8070fe0e89eeb230f3dcfc032dad2b213"} Oct 30 14:58:11.758620 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:11.758591 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" event={"ID":"72284b4e-d5b0-4317-b86f-7341f3d3364f","Type":"ContainerStarted","Data":"dfffa621c7e9cfb0339a5dbe9adc0733123e25577b40f158fec4987402b59ea4"} Oct 30 14:58:11.809459 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:11.809403 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podStartSLOduration=1.5493190669999999 podStartE2EDuration="1.809387907s" totalImagesPullingTime="260.06884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:11.808310078 +0000 UTC m=+595.767351485" watchObservedRunningTime="2026-10-30 14:58:11.809387907 +0000 UTC m=+595.768429315" Oct 30 14:58:11.872263 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:11.872235 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:13.425656 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:13.425625 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:13.438209 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:13.438183 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:13.779668 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:13.779590 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-pspnq" Oct 30 14:58:17.964442 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:17.964406 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:17.976657 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:17.976636 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:18.802494 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:18.802460 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn" Oct 30 14:58:20.872007 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:20.871974 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:20.883942 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:20.883921 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:21.815159 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:21.815131 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 14:58:24.645160 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.645134 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7ff567df79-mrmbk"] Oct 30 14:58:24.648629 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.648613 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.668808 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.668783 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff567df79-mrmbk"] Oct 30 14:58:24.754480 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.754454 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgzlv\" (UniqueName: \"kubernetes.io/projected/d0ef798f-d74c-4d9a-8c55-c40ed34976a9-kube-api-access-bgzlv\") pod \"authorino-7ff567df79-mrmbk\" (UID: \"d0ef798f-d74c-4d9a-8c55-c40ed34976a9\") " pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.754597 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.754486 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d0ef798f-d74c-4d9a-8c55-c40ed34976a9-tls-cert\") pod \"authorino-7ff567df79-mrmbk\" (UID: \"d0ef798f-d74c-4d9a-8c55-c40ed34976a9\") " pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.855426 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.855406 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d0ef798f-d74c-4d9a-8c55-c40ed34976a9-tls-cert\") pod \"authorino-7ff567df79-mrmbk\" (UID: \"d0ef798f-d74c-4d9a-8c55-c40ed34976a9\") " pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.855529 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.855479 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgzlv\" (UniqueName: \"kubernetes.io/projected/d0ef798f-d74c-4d9a-8c55-c40ed34976a9-kube-api-access-bgzlv\") pod \"authorino-7ff567df79-mrmbk\" (UID: \"d0ef798f-d74c-4d9a-8c55-c40ed34976a9\") " pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.857698 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.857676 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d0ef798f-d74c-4d9a-8c55-c40ed34976a9-tls-cert\") pod \"authorino-7ff567df79-mrmbk\" (UID: \"d0ef798f-d74c-4d9a-8c55-c40ed34976a9\") " pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.880076 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.880050 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgzlv\" (UniqueName: \"kubernetes.io/projected/d0ef798f-d74c-4d9a-8c55-c40ed34976a9-kube-api-access-bgzlv\") pod \"authorino-7ff567df79-mrmbk\" (UID: \"d0ef798f-d74c-4d9a-8c55-c40ed34976a9\") " pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:24.959728 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:24.959679 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7ff567df79-mrmbk" Oct 30 14:58:25.255220 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:25.255196 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7ff567df79-mrmbk"] Oct 30 14:58:25.257573 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:25.257544 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0ef798f_d74c_4d9a_8c55_c40ed34976a9.slice/crio-f720cc4c2e92848e803131ec181402b9ee254d02b8c8b187d1638fb0600424c9 WatchSource:0}: Error finding container f720cc4c2e92848e803131ec181402b9ee254d02b8c8b187d1638fb0600424c9: Status 404 returned error can't find the container with id f720cc4c2e92848e803131ec181402b9ee254d02b8c8b187d1638fb0600424c9 Oct 30 14:58:25.822914 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:25.822883 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff567df79-mrmbk" event={"ID":"d0ef798f-d74c-4d9a-8c55-c40ed34976a9","Type":"ContainerStarted","Data":"f720cc4c2e92848e803131ec181402b9ee254d02b8c8b187d1638fb0600424c9"} Oct 30 14:58:26.829243 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:26.829197 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7ff567df79-mrmbk" event={"ID":"d0ef798f-d74c-4d9a-8c55-c40ed34976a9","Type":"ContainerStarted","Data":"258cc2d442b9efabb9238d0a5c5c221c34b819e56301ea8b7aab9a4d5d81cf66"} Oct 30 14:58:26.885748 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:26.885681 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7ff567df79-mrmbk" podStartSLOduration=2.172115825 podStartE2EDuration="2.885661152s" totalImagesPullingTime="713.545327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:26.885168566 +0000 UTC m=+610.844209971" watchObservedRunningTime="2026-10-30 14:58:26.885661152 +0000 UTC m=+610.844702558" Oct 30 14:58:26.904796 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:26.904751 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77cffb447b-5qfb5"] Oct 30 14:58:26.905050 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:26.905020 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-77cffb447b-5qfb5" podUID="464ec6bb-108c-41fe-accd-83589d9ec50e" containerName="authorino" containerID="cri-o://23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610" gracePeriod=30 Oct 30 14:58:27.150162 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.150140 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:58:27.291422 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.291383 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert\" (UniqueName: \"kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert\") pod \"464ec6bb-108c-41fe-accd-83589d9ec50e\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " Oct 30 14:58:27.291608 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.291474 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc\") pod \"464ec6bb-108c-41fe-accd-83589d9ec50e\" (UID: \"464ec6bb-108c-41fe-accd-83589d9ec50e\") " Oct 30 14:58:27.293498 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.293462 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc" pod "464ec6bb-108c-41fe-accd-83589d9ec50e" (UID: "464ec6bb-108c-41fe-accd-83589d9ec50e"). InnerVolumeSpecName "kube-api-access-t45rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 14:58:27.301551 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.301522 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert" pod "464ec6bb-108c-41fe-accd-83589d9ec50e" (UID: "464ec6bb-108c-41fe-accd-83589d9ec50e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 14:58:27.392992 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.392965 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t45rc\" (UniqueName: \"kubernetes.io/projected/464ec6bb-108c-41fe-accd-83589d9ec50e-kube-api-access-t45rc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:58:27.392992 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.392993 6554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/464ec6bb-108c-41fe-accd-83589d9ec50e-tls-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 14:58:27.834653 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.834571 6554 generic.go:356] "Generic (PLEG): container finished" podID="464ec6bb-108c-41fe-accd-83589d9ec50e" containerID="23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610" exitCode=0 Oct 30 14:58:27.835137 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.835119 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77cffb447b-5qfb5" Oct 30 14:58:27.836457 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.836421 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cffb447b-5qfb5" event={"ID":"464ec6bb-108c-41fe-accd-83589d9ec50e","Type":"ContainerDied","Data":"23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610"} Oct 30 14:58:27.836584 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.836486 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77cffb447b-5qfb5" event={"ID":"464ec6bb-108c-41fe-accd-83589d9ec50e","Type":"ContainerDied","Data":"2bf8f055d8f88f00818848183fd018d4a3931c90dc17d0f6d5eabc1ac3664134"} Oct 30 14:58:27.836584 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.836506 6554 scope.go:122] "RemoveContainer" containerID="23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610" Oct 30 14:58:27.845256 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.845234 6554 scope.go:122] "RemoveContainer" containerID="23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610" Oct 30 14:58:27.846044 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:58:27.846023 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610\": container with ID starting with 23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610 not found: ID does not exist" containerID="23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610" Oct 30 14:58:27.846108 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.846054 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610"} err="failed to get container status \"23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610\": rpc error: code = NotFound desc = could not find container \"23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610\": container with ID starting with 23e09eff155b32e53770b73b739d854fc2d4b63d96a3ddbcc69e662045f1f610 not found: ID does not exist" Oct 30 14:58:27.892222 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.892194 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-77cffb447b-5qfb5"] Oct 30 14:58:27.905175 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:27.905155 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-77cffb447b-5qfb5"] Oct 30 14:58:28.152903 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:28.152871 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464ec6bb-108c-41fe-accd-83589d9ec50e" path="/var/lib/kubelet/pods/464ec6bb-108c-41fe-accd-83589d9ec50e/volumes" Oct 30 14:58:35.388520 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.388481 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6"] Oct 30 14:58:35.389036 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.388843 6554 cpu_manager.go:414] "containerMap: removing container" podUID="464ec6bb-108c-41fe-accd-83589d9ec50e" containerName="authorino" Oct 30 14:58:35.389036 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.388855 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464ec6bb-108c-41fe-accd-83589d9ec50e" containerName="authorino" Oct 30 14:58:35.389036 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.388930 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="464ec6bb-108c-41fe-accd-83589d9ec50e" containerName="authorino" Oct 30 14:58:35.394343 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.394321 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.412854 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.412832 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 14:58:35.425284 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.425260 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khr2k\"" Oct 30 14:58:35.426888 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.426866 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6"] Oct 30 14:58:35.482364 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482323 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482460 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482446 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg2l\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482509 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482476 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482509 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482502 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482709 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482688 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482842 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482813 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482962 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482849 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.482962 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482878 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.483046 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.482970 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584109 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584079 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584219 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584124 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584219 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584175 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584467 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584444 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg2l\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584562 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584481 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584562 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584502 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584679 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584600 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584679 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584617 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584679 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584643 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584835 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584705 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.584917 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584894 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.585022 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.584998 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.585102 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.585061 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.585271 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.585244 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.586874 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.586851 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.587137 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.587118 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.608899 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.608876 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.609210 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.609192 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg2l\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l\") pod \"maas-default-gateway-openshift-default-6cc458d887-bh8r6\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:35.709008 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:35.708957 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:36.025215 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:36.025191 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6"] Oct 30 14:58:36.027185 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:36.027154 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5a9032_6fb8_4833_992c_e128a4419f73.slice/crio-da320cc10e96c7313e0df731291aa7e6f12376b73514c46c2acfe965c6b8340e WatchSource:0}: Error finding container da320cc10e96c7313e0df731291aa7e6f12376b73514c46c2acfe965c6b8340e: Status 404 returned error can't find the container with id da320cc10e96c7313e0df731291aa7e6f12376b73514c46c2acfe965c6b8340e Oct 30 14:58:36.878747 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:36.878706 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" event={"ID":"0b5a9032-6fb8-4833-992c-e128a4419f73","Type":"ContainerStarted","Data":"da320cc10e96c7313e0df731291aa7e6f12376b73514c46c2acfe965c6b8340e"} Oct 30 14:58:38.636085 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:38.636054 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:58:38.636359 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:38.636121 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:58:38.636359 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:38.636183 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:58:38.891341 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:38.891309 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" event={"ID":"0b5a9032-6fb8-4833-992c-e128a4419f73","Type":"ContainerStarted","Data":"b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544"} Oct 30 14:58:38.954585 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:38.954531 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podStartSLOduration=1.348353993 podStartE2EDuration="3.954514888s" totalImagesPullingTime="2.606160895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:38.953724357 +0000 UTC m=+622.912765755" watchObservedRunningTime="2026-10-30 14:58:38.954514888 +0000 UTC m=+622.913556294" Oct 30 14:58:39.202096 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:58:39.202011 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:58:39.709987 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:39.709954 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:39.711629 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:39.711596 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:39.711753 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:39.711666 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:40.709835 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:40.709800 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:40.710015 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:40.709862 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:41.709832 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:41.709794 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:41.710011 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:41.709851 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:42.709893 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:42.709856 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:42.710294 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:42.709910 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:43.710003 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:43.709970 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:43.710419 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:43.710025 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:44.710142 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:44.710104 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:44.710562 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:44.710170 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:45.709704 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:45.709672 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-bh8r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 14:58:45.709873 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:45.709722 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 14:58:46.847835 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:46.847805 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:46.848323 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:46.848201 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:46.849238 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:46.849217 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 14:58:50.422663 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.422627 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8"] Oct 30 14:58:50.427371 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.427328 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.460776 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.460750 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8"] Oct 30 14:58:50.530910 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.530880 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531048 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.530917 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531048 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.530944 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531160 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.531061 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531160 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.531083 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531160 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.531111 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531256 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.531219 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531256 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.531248 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.531324 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.531283 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl2k5\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.631875 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.631845 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.631998 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.631881 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.631998 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.631902 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632087 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632024 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632087 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632071 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632099 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632156 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl2k5\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632320 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632291 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632320 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632310 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632463 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632395 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632463 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632403 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632463 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632452 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632625 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632576 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.632889 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.632868 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.634554 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.634521 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.634941 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.634922 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.657136 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.657112 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.657621 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.657592 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl2k5\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5\") pod \"maas-default-gateway-openshift-default-6cc458d887-tw6z8\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:50.739642 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:50.739594 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:51.054668 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:51.054614 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8"] Oct 30 14:58:51.057372 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:58:51.057329 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996e7f9d_6aee_4621_9f6b_10892c43f243.slice/crio-586cca9aa35ec6aaf603c024759ed7594a458089c8720105cad4d44b34ab298f WatchSource:0}: Error finding container 586cca9aa35ec6aaf603c024759ed7594a458089c8720105cad4d44b34ab298f: Status 404 returned error can't find the container with id 586cca9aa35ec6aaf603c024759ed7594a458089c8720105cad4d44b34ab298f Oct 30 14:58:51.059395 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:51.059359 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:58:51.059468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:51.059422 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:58:51.059468 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:51.059455 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:58:51.951953 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:51.951919 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" event={"ID":"996e7f9d-6aee-4621-9f6b-10892c43f243","Type":"ContainerStarted","Data":"b7ed02841903c5f181f69b737849bca474a5ff8687e2dc1241f8a59a2df93634"} Oct 30 14:58:51.952427 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:51.951976 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" event={"ID":"996e7f9d-6aee-4621-9f6b-10892c43f243","Type":"ContainerStarted","Data":"586cca9aa35ec6aaf603c024759ed7594a458089c8720105cad4d44b34ab298f"} Oct 30 14:58:52.015774 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:52.015718 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podStartSLOduration=2.015705265 podStartE2EDuration="2.015705265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:58:52.01467386 +0000 UTC m=+635.973715278" watchObservedRunningTime="2026-10-30 14:58:52.015705265 +0000 UTC m=+635.974746670" Oct 30 14:58:52.740753 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:52.740727 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:52.742012 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:52.741992 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tw6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:58:52.742083 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:52.742036 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 14:58:53.740667 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:53.740630 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tw6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:58:53.741034 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:53.740694 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 14:58:54.740611 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:54.740579 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tw6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:58:54.740776 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:54.740638 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 14:58:55.740501 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:55.740472 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tw6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:58:55.740707 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:55.740529 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 14:58:56.740496 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:56.740459 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tw6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:58:56.740984 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:56.740518 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 14:58:57.740090 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:57.740052 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-tw6z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 14:58:57.740241 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:57.740102 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 14:58:59.269191 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:59.269161 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:59.269670 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:59.269511 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:58:59.270386 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:58:59.270365 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 14:59:09.199743 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:59:09.199706 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:59:23.340926 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.340893 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9"] Oct 30 14:59:23.350654 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.350626 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.370325 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.370303 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-357948-openshift-default-dockercfg-b8jx8\"" Oct 30 14:59:23.384856 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.384831 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9"] Oct 30 14:59:23.431667 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.431644 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.431783 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.431697 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.431837 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.431796 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.431837 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.431830 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.431977 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.431952 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.432036 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.432008 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.432244 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.432077 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.432244 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.432129 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.432244 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.432168 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cg9\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.532986 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.532947 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533100 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533025 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533100 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533062 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533194 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533119 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533194 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533158 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cg9\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533296 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533196 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533296 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533244 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533296 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533276 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533306 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533482 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533460 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533724 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533697 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533841 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533731 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.533841 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.533800 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.534052 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.534031 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.535389 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.535343 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.535812 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.535797 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.558411 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.558387 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.558536 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.558513 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cg9\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.663293 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.663267 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:23.690581 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.690558 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs"] Oct 30 14:59:23.696001 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.695972 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.716229 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.716199 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-05d396-openshift-default-dockercfg-rlrzr\"" Oct 30 14:59:23.731858 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.727753 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs"] Oct 30 14:59:23.839012 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.838981 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s"] Oct 30 14:59:23.839512 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839485 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839524 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839555 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839572 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839599 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839629 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839915 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839654 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839915 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839676 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.839915 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.839704 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8h7s\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.843440 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.843419 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.882043 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.882019 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s"] Oct 30 14:59:23.922012 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.921953 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4b703c-openshift-default-dockercfg-f85tr\"" Oct 30 14:59:23.940381 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940330 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940381 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940373 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940548 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940396 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.940548 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940416 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284zb\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.940548 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940438 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940548 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940454 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.940548 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940480 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940804 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940702 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.940804 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940747 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940804 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940749 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940804 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940754 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940980 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940826 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940980 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940862 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.940980 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940886 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940980 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940904 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8h7s\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.940980 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.940946 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.941198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941008 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.941198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941038 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.941198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941043 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.941198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941106 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:23.941198 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941150 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.941421 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941223 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.941421 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.941251 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.942901 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.942871 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:23.943490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:23.943474 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:24.006101 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.006070 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8h7s\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:24.011523 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.011499 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:24.013447 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.013430 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:24.042116 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042088 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042248 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042147 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042248 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042191 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042384 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042250 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042384 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042279 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042384 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042307 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042415 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042447 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284zb\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042550 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042489 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042810 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042785 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.042876 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.042822 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.043333 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.043315 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.043439 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.043404 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.043504 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.043436 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.044595 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.044570 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.045130 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.045111 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.058014 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.056627 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9"] Oct 30 14:59:24.059888 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:24.059865 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb980b704_8bd8_46ce_b9ee_de7ddc035ec5.slice/crio-5bf72c3fad2d46a3cc7c3205c17b9f7d1c2c4ce1d6c6f2dc6b0867ab1c2ca313 WatchSource:0}: Error finding container 5bf72c3fad2d46a3cc7c3205c17b9f7d1c2c4ce1d6c6f2dc6b0867ab1c2ca313: Status 404 returned error can't find the container with id 5bf72c3fad2d46a3cc7c3205c17b9f7d1c2c4ce1d6c6f2dc6b0867ab1c2ca313 Oct 30 14:59:24.062105 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.062053 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.062211 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.062138 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.062211 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.062177 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.073444 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.073415 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.075839 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.075815 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284zb\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.122641 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.122612 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" event={"ID":"b980b704-8bd8-46ce-b9ee-de7ddc035ec5","Type":"ContainerStarted","Data":"5bf72c3fad2d46a3cc7c3205c17b9f7d1c2c4ce1d6c6f2dc6b0867ab1c2ca313"} Oct 30 14:59:24.159640 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.159607 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:24.336886 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.336850 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs"] Oct 30 14:59:24.339830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.339439 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.339939 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.339895 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.339998 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.339951 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.470581 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.470549 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s"] Oct 30 14:59:24.471850 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:24.471820 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aab4ca9_aa15_4ea5_9919_6585d9bf6718.slice/crio-9faafdf238a0fa9a4ae1fbcc318798348f19ed3c2b2556d006368c6348542a2c WatchSource:0}: Error finding container 9faafdf238a0fa9a4ae1fbcc318798348f19ed3c2b2556d006368c6348542a2c: Status 404 returned error can't find the container with id 9faafdf238a0fa9a4ae1fbcc318798348f19ed3c2b2556d006368c6348542a2c Oct 30 14:59:24.474445 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.474414 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.474526 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.474476 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:24.474526 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:24.474509 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:25.130638 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.130600 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" event={"ID":"b85a8334-2fe2-4991-86b4-d4961aefb039","Type":"ContainerStarted","Data":"f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c"} Oct 30 14:59:25.130822 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.130664 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" event={"ID":"b85a8334-2fe2-4991-86b4-d4961aefb039","Type":"ContainerStarted","Data":"de71fd69f32a808437d4e8b8a920bd03638ba8f930e063c29dff6b865b308f4f"} Oct 30 14:59:25.132308 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.132282 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" event={"ID":"b980b704-8bd8-46ce-b9ee-de7ddc035ec5","Type":"ContainerStarted","Data":"1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63"} Oct 30 14:59:25.133751 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.133730 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" event={"ID":"1aab4ca9-aa15-4ea5-9919-6585d9bf6718","Type":"ContainerStarted","Data":"4447ccfff210a25a263753a91d4f737cd118225ba604a880ed746bf9b5c6c19d"} Oct 30 14:59:25.133840 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.133773 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" event={"ID":"1aab4ca9-aa15-4ea5-9919-6585d9bf6718","Type":"ContainerStarted","Data":"9faafdf238a0fa9a4ae1fbcc318798348f19ed3c2b2556d006368c6348542a2c"} Oct 30 14:59:25.160131 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.160105 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:25.164762 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.164743 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:25.179256 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.179206 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podStartSLOduration=2.17918978 podStartE2EDuration="2.17918978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:25.178794088 +0000 UTC m=+669.137835494" watchObservedRunningTime="2026-10-30 14:59:25.17918978 +0000 UTC m=+669.138231199" Oct 30 14:59:25.235219 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.235178 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podStartSLOduration=2.23516658 podStartE2EDuration="2.23516658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:25.234131278 +0000 UTC m=+669.193172685" watchObservedRunningTime="2026-10-30 14:59:25.23516658 +0000 UTC m=+669.194207985" Oct 30 14:59:25.283514 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.283481 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podStartSLOduration=2.283471625 podStartE2EDuration="2.283471625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:25.281940351 +0000 UTC m=+669.240981755" watchObservedRunningTime="2026-10-30 14:59:25.283471625 +0000 UTC m=+669.242513031" Oct 30 14:59:25.664336 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.664305 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:25.694404 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:25.694343 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:26.013966 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.013898 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:26.018738 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.018714 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:26.139188 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.139005 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:26.139188 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.139068 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:26.139188 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.139091 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:26.139753 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.139726 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:26.139884 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.139866 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 14:59:26.140470 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:26.140454 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:29.075199 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.075167 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv"] Oct 30 14:59:29.078773 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.078756 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn"] Oct 30 14:59:29.078888 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.078868 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.082058 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.082037 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.102295 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.102276 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e64e05\"" Oct 30 14:59:29.102295 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.102287 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e64e05-dockercfg-qgkvj\"" Oct 30 14:59:29.116487 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.116466 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn"] Oct 30 14:59:29.118039 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.118007 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv"] Oct 30 14:59:29.193771 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.193738 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.193967 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.193803 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6l4r\" (UniqueName: \"kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r\") pod \"payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.193967 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.193886 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglmc\" (UniqueName: \"kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc\") pod \"payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.193967 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.193924 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.294816 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.294785 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.294935 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.294851 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6l4r\" (UniqueName: \"kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r\") pod \"payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.294935 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.294894 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xglmc\" (UniqueName: \"kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc\") pod \"payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.294935 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.294919 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.295772 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.295751 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.295828 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.295751 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.321199 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.321173 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6l4r\" (UniqueName: \"kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r\") pod \"payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.321497 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.321478 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglmc\" (UniqueName: \"kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc\") pod \"payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.394051 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.394028 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:29.399887 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.399869 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:29.704536 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.704512 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv"] Oct 30 14:59:29.706992 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:29.706963 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf683f8ad_7e91_49a1_abe4_6b099388fc81.slice/crio-d1c580919f7a4ab1e6a6ef736a6e0317b0505d24e81e2c66f602edced326f649 WatchSource:0}: Error finding container d1c580919f7a4ab1e6a6ef736a6e0317b0505d24e81e2c66f602edced326f649: Status 404 returned error can't find the container with id d1c580919f7a4ab1e6a6ef736a6e0317b0505d24e81e2c66f602edced326f649 Oct 30 14:59:29.745742 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:29.745718 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn"] Oct 30 14:59:29.748589 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:29.748565 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e069241_0830_4dbe_8fba_f1e867496dbb.slice/crio-d909860d3b6c595d47e43e4504f2ca74ef3477cf87ff6126da6dad372d620fe1 WatchSource:0}: Error finding container d909860d3b6c595d47e43e4504f2ca74ef3477cf87ff6126da6dad372d620fe1: Status 404 returned error can't find the container with id d909860d3b6c595d47e43e4504f2ca74ef3477cf87ff6126da6dad372d620fe1 Oct 30 14:59:30.161332 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:30.161293 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" event={"ID":"2e069241-0830-4dbe-8fba-f1e867496dbb","Type":"ContainerStarted","Data":"d909860d3b6c595d47e43e4504f2ca74ef3477cf87ff6126da6dad372d620fe1"} Oct 30 14:59:30.164031 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:30.164001 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" event={"ID":"f683f8ad-7e91-49a1-abe4-6b099388fc81","Type":"ContainerStarted","Data":"d1c580919f7a4ab1e6a6ef736a6e0317b0505d24e81e2c66f602edced326f649"} Oct 30 14:59:31.070834 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.070795 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46"] Oct 30 14:59:31.078066 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.077932 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.099003 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.098950 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46"] Oct 30 14:59:31.112413 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.112389 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-4f5c30\"" Oct 30 14:59:31.112633 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.112617 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-4f5c30-dockercfg-47svg\"" Oct 30 14:59:31.216509 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.216476 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.216966 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.216599 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66d8s\" (UniqueName: \"kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s\") pod \"payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.318204 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.318166 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66d8s\" (UniqueName: \"kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s\") pod \"payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.318417 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.318245 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.319297 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.319269 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.352627 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.352504 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66d8s\" (UniqueName: \"kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s\") pod \"payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.403755 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.403722 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:31.715878 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:31.715844 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46"] Oct 30 14:59:31.719580 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:31.719546 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101b3da3_6528_4ec3_9c33_992975d2c884.slice/crio-4022a0041bba1fe6093881fddf18309de048f6de0231adb3f544f78e3939e8fa WatchSource:0}: Error finding container 4022a0041bba1fe6093881fddf18309de048f6de0231adb3f544f78e3939e8fa: Status 404 returned error can't find the container with id 4022a0041bba1fe6093881fddf18309de048f6de0231adb3f544f78e3939e8fa Oct 30 14:59:32.174653 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:32.174627 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" event={"ID":"101b3da3-6528-4ec3-9c33-992975d2c884","Type":"ContainerStarted","Data":"4022a0041bba1fe6093881fddf18309de048f6de0231adb3f544f78e3939e8fa"} Oct 30 14:59:33.181593 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.181553 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" event={"ID":"101b3da3-6528-4ec3-9c33-992975d2c884","Type":"ContainerStarted","Data":"bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f"} Oct 30 14:59:33.182059 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.181736 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:33.183215 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.183191 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" event={"ID":"2e069241-0830-4dbe-8fba-f1e867496dbb","Type":"ContainerStarted","Data":"57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134"} Oct 30 14:59:33.183331 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.183275 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:33.184774 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.184756 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" event={"ID":"f683f8ad-7e91-49a1-abe4-6b099388fc81","Type":"ContainerStarted","Data":"361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba"} Oct 30 14:59:33.184891 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.184879 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:33.228142 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.228091 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" podStartSLOduration=1.8163065390000002 podStartE2EDuration="2.228076785s" totalImagesPullingTime="411.770246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:33.226280541 +0000 UTC m=+677.185321974" watchObservedRunningTime="2026-10-30 14:59:33.228076785 +0000 UTC m=+677.187118183" Oct 30 14:59:33.271739 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.271680 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" podStartSLOduration=1.846525588 podStartE2EDuration="4.271663808s" totalImagesPullingTime="2.42513822s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:33.271519609 +0000 UTC m=+677.230561016" watchObservedRunningTime="2026-10-30 14:59:33.271663808 +0000 UTC m=+677.230705212" Oct 30 14:59:33.320056 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.320006 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" podStartSLOduration=1.923832107 podStartE2EDuration="4.319989866s" totalImagesPullingTime="2.396157759s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:33.318896901 +0000 UTC m=+677.277938308" watchObservedRunningTime="2026-10-30 14:59:33.319989866 +0000 UTC m=+677.279031289" Oct 30 14:59:33.540169 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.540088 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d"] Oct 30 14:59:33.543939 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.543920 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.545578 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.545553 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.545761 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.545746 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xstf\" (UniqueName: \"kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf\") pod \"payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.561053 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.561035 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-357948-dockercfg-94mbp\"" Oct 30 14:59:33.568341 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.568317 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d"] Oct 30 14:59:33.570505 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.570489 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-357948\"" Oct 30 14:59:33.626933 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.624889 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f"] Oct 30 14:59:33.640753 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.640715 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.646473 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.646443 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f"] Oct 30 14:59:33.648019 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.647980 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.648137 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.648076 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hrxk\" (UniqueName: \"kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk\") pod \"payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.648182 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.648145 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.648231 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.648177 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xstf\" (UniqueName: \"kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf\") pod \"payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.649295 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.649268 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.676705 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.676674 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xstf\" (UniqueName: \"kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf\") pod \"payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.749115 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.749085 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hrxk\" (UniqueName: \"kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk\") pod \"payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.749274 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.749143 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.749788 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.749770 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.777142 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.777098 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hrxk\" (UniqueName: \"kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk\") pod \"payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:33.860260 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.860197 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:33.963853 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:33.963822 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:34.158431 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:34.158388 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d"] Oct 30 14:59:34.160544 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:34.160521 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629da890_dde2_471f_a304_acf45a058a8d.slice/crio-89fe5b78f51087497b7d32057c8f7261f012364fcc9e456363e9c6492d0308ac WatchSource:0}: Error finding container 89fe5b78f51087497b7d32057c8f7261f012364fcc9e456363e9c6492d0308ac: Status 404 returned error can't find the container with id 89fe5b78f51087497b7d32057c8f7261f012364fcc9e456363e9c6492d0308ac Oct 30 14:59:34.201423 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:34.201393 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" event={"ID":"629da890-dde2-471f-a304-acf45a058a8d","Type":"ContainerStarted","Data":"89fe5b78f51087497b7d32057c8f7261f012364fcc9e456363e9c6492d0308ac"} Oct 30 14:59:34.266892 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:34.266787 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f"] Oct 30 14:59:34.269495 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:34.269469 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fd997d_2b65_49b4_a644_f1438a37c184.slice/crio-e95b23e04b8a9ab18d1f928f7ee29929c3ad43efb6a3531fc679bcfc0ab278aa WatchSource:0}: Error finding container e95b23e04b8a9ab18d1f928f7ee29929c3ad43efb6a3531fc679bcfc0ab278aa: Status 404 returned error can't find the container with id e95b23e04b8a9ab18d1f928f7ee29929c3ad43efb6a3531fc679bcfc0ab278aa Oct 30 14:59:35.207642 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.207603 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" event={"ID":"629da890-dde2-471f-a304-acf45a058a8d","Type":"ContainerStarted","Data":"644a5c9b17c6b87d6f4886b246ea7d72a2ed65ee4de9b468350c022d17db7680"} Oct 30 14:59:35.208127 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.207673 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:35.209061 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.209034 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" event={"ID":"f4fd997d-2b65-49b4-a644-f1438a37c184","Type":"ContainerStarted","Data":"b6aaef3402c9c24f6cbd9e78e8948206e5c03ea052bec03e691b5dc80581448b"} Oct 30 14:59:35.209178 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.209080 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" event={"ID":"f4fd997d-2b65-49b4-a644-f1438a37c184","Type":"ContainerStarted","Data":"e95b23e04b8a9ab18d1f928f7ee29929c3ad43efb6a3531fc679bcfc0ab278aa"} Oct 30 14:59:35.209178 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.209124 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:35.254332 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.254290 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" podStartSLOduration=2.254277682 podStartE2EDuration="2.254277682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:35.253568966 +0000 UTC m=+679.212610371" watchObservedRunningTime="2026-10-30 14:59:35.254277682 +0000 UTC m=+679.213319099" Oct 30 14:59:35.301369 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.301286 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" podStartSLOduration=2.301273645 podStartE2EDuration="2.301273645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:35.299941639 +0000 UTC m=+679.258983045" watchObservedRunningTime="2026-10-30 14:59:35.301273645 +0000 UTC m=+679.260315049" Oct 30 14:59:35.516659 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.516600 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z"] Oct 30 14:59:35.520313 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.520289 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.540324 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.540300 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z"] Oct 30 14:59:35.540993 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.540970 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-2c8d89-dockercfg-dktlb\"" Oct 30 14:59:35.613050 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.613025 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-2c8d89\"" Oct 30 14:59:35.667216 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.667189 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn9t2\" (UniqueName: \"kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2\") pod \"payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.667378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.667227 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.767767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.767704 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn9t2\" (UniqueName: \"kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2\") pod \"payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.767767 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.767743 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.768465 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.768444 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.793650 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.793622 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn9t2\" (UniqueName: \"kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2\") pod \"payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:35.833810 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:35.833783 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:36.130612 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:36.130585 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z"] Oct 30 14:59:36.132748 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:36.132720 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9ba3eb_543e_436e_8de8_19ccff16926a.slice/crio-33ea7e5a1b11efe74232912d57d7cdb10050871c06754a3a3f405fd77774fd1d WatchSource:0}: Error finding container 33ea7e5a1b11efe74232912d57d7cdb10050871c06754a3a3f405fd77774fd1d: Status 404 returned error can't find the container with id 33ea7e5a1b11efe74232912d57d7cdb10050871c06754a3a3f405fd77774fd1d Oct 30 14:59:36.216291 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:36.216261 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" event={"ID":"7a9ba3eb-543e-436e-8de8-19ccff16926a","Type":"ContainerStarted","Data":"94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02"} Oct 30 14:59:36.216760 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:36.216319 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" event={"ID":"7a9ba3eb-543e-436e-8de8-19ccff16926a","Type":"ContainerStarted","Data":"33ea7e5a1b11efe74232912d57d7cdb10050871c06754a3a3f405fd77774fd1d"} Oct 30 14:59:36.216760 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:36.216611 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:36.260236 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:36.260192 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" podStartSLOduration=1.260181041 podStartE2EDuration="1.260181041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:36.258832226 +0000 UTC m=+680.217873652" watchObservedRunningTime="2026-10-30 14:59:36.260181041 +0000 UTC m=+680.219222445" Oct 30 14:59:38.120594 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.120560 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x"] Oct 30 14:59:38.124283 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.124263 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.159480 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.159457 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-83ff4e\"" Oct 30 14:59:38.159607 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.159586 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-83ff4e-dockercfg-flrr7\"" Oct 30 14:59:38.161037 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.161019 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x"] Oct 30 14:59:38.172618 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.172592 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts"] Oct 30 14:59:38.176049 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.176029 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.191953 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.191933 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5rpm\" (UniqueName: \"kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm\") pod \"payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.192058 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.191992 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.192138 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.192118 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974zp\" (UniqueName: \"kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp\") pod \"payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.192182 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.192154 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.197069 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.197039 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts"] Oct 30 14:59:38.293305 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.293273 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5rpm\" (UniqueName: \"kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm\") pod \"payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.293489 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.293325 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.293489 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.293412 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974zp\" (UniqueName: \"kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp\") pod \"payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.293489 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.293440 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.294189 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.294161 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.294281 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.294171 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.318218 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.318191 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974zp\" (UniqueName: \"kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp\") pod \"payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.318374 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.318330 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5rpm\" (UniqueName: \"kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm\") pod \"payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.436380 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.436337 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:38.492770 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.492739 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:38.738227 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.738204 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x"] Oct 30 14:59:38.740195 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:38.740169 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87357b67_8af1_423f_b92c_5370575b46e1.slice/crio-3d5a67e8f8058505b1a936dc295060ad20267afb3100be1159d69c7448c8f2cc WatchSource:0}: Error finding container 3d5a67e8f8058505b1a936dc295060ad20267afb3100be1159d69c7448c8f2cc: Status 404 returned error can't find the container with id 3d5a67e8f8058505b1a936dc295060ad20267afb3100be1159d69c7448c8f2cc Oct 30 14:59:38.798665 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:38.798639 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts"] Oct 30 14:59:38.800759 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:38.800734 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd69746aa_ea97_4e2e_ae58_22716e1c8160.slice/crio-0d19036a499f2103e877346ccd7179c8808f1a51e20fb09a7d0e87e44dd83313 WatchSource:0}: Error finding container 0d19036a499f2103e877346ccd7179c8808f1a51e20fb09a7d0e87e44dd83313: Status 404 returned error can't find the container with id 0d19036a499f2103e877346ccd7179c8808f1a51e20fb09a7d0e87e44dd83313 Oct 30 14:59:39.200272 ip-10-0-139-204 kubenswrapper[6554]: E1030 14:59:39.200238 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 14:59:39.234869 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.234838 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" event={"ID":"d69746aa-ea97-4e2e-ae58-22716e1c8160","Type":"ContainerStarted","Data":"110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f"} Oct 30 14:59:39.235004 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.234903 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:39.235004 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.234930 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" event={"ID":"d69746aa-ea97-4e2e-ae58-22716e1c8160","Type":"ContainerStarted","Data":"0d19036a499f2103e877346ccd7179c8808f1a51e20fb09a7d0e87e44dd83313"} Oct 30 14:59:39.236419 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.236394 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" event={"ID":"87357b67-8af1-423f-b92c-5370575b46e1","Type":"ContainerStarted","Data":"12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd"} Oct 30 14:59:39.236508 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.236443 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" event={"ID":"87357b67-8af1-423f-b92c-5370575b46e1","Type":"ContainerStarted","Data":"3d5a67e8f8058505b1a936dc295060ad20267afb3100be1159d69c7448c8f2cc"} Oct 30 14:59:39.236508 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.236473 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:39.280210 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.280163 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" podStartSLOduration=1.2801480760000001 podStartE2EDuration="1.280148076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:39.27815982 +0000 UTC m=+683.237201436" watchObservedRunningTime="2026-10-30 14:59:39.280148076 +0000 UTC m=+683.239189512" Oct 30 14:59:39.327228 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:39.327173 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" podStartSLOduration=1.327157661 podStartE2EDuration="1.327157661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:39.326220765 +0000 UTC m=+683.285262169" watchObservedRunningTime="2026-10-30 14:59:39.327157661 +0000 UTC m=+683.286199067" Oct 30 14:59:41.107452 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.107412 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m"] Oct 30 14:59:41.112367 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.112333 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.147511 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.147479 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m"] Oct 30 14:59:41.158409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.158379 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4b703c-dockercfg-cnvf5\"" Oct 30 14:59:41.158409 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.158404 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4b703c\"" Oct 30 14:59:41.173589 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.173561 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc"] Oct 30 14:59:41.178262 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.178236 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.216367 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.215979 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc"] Oct 30 14:59:41.220574 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.220532 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.220718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.220583 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzb9\" (UniqueName: \"kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9\") pod \"payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.220718 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.220658 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.220849 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.220764 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbqb\" (UniqueName: \"kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb\") pod \"payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.321933 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.321877 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.322097 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.321955 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzb9\" (UniqueName: \"kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9\") pod \"payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.322097 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.322027 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.322182 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.322167 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbqb\" (UniqueName: \"kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb\") pod \"payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.322864 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.322841 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.322909 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.322841 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.388418 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.372169 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzb9\" (UniqueName: \"kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9\") pod \"payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.396378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.392991 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbqb\" (UniqueName: \"kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb\") pod \"payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.437381 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.437322 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:41.492962 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.492926 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:41.767659 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.767638 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m"] Oct 30 14:59:41.769833 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:41.769804 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3571d646_2915_4c39_8db1_e35152a7027d.slice/crio-05593a128dbd6e4663ecc3f797d95d30491ef05cf741900c9d8d45ac334e9d0f WatchSource:0}: Error finding container 05593a128dbd6e4663ecc3f797d95d30491ef05cf741900c9d8d45ac334e9d0f: Status 404 returned error can't find the container with id 05593a128dbd6e4663ecc3f797d95d30491ef05cf741900c9d8d45ac334e9d0f Oct 30 14:59:41.828025 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:41.828003 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc"] Oct 30 14:59:41.828678 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:41.828656 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf767f3a1_2ee7_47eb_bb0d_d634b0f3ffa5.slice/crio-27447ee701a8428fc1d27c2fce3048d646e1dc7b91adbec6f985bc917080978e WatchSource:0}: Error finding container 27447ee701a8428fc1d27c2fce3048d646e1dc7b91adbec6f985bc917080978e: Status 404 returned error can't find the container with id 27447ee701a8428fc1d27c2fce3048d646e1dc7b91adbec6f985bc917080978e Oct 30 14:59:42.276372 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.273133 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" event={"ID":"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5","Type":"ContainerStarted","Data":"5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b"} Oct 30 14:59:42.276372 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.273201 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" event={"ID":"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5","Type":"ContainerStarted","Data":"27447ee701a8428fc1d27c2fce3048d646e1dc7b91adbec6f985bc917080978e"} Oct 30 14:59:42.276372 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.274118 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:42.279204 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.279173 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" event={"ID":"3571d646-2915-4c39-8db1-e35152a7027d","Type":"ContainerStarted","Data":"4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc"} Oct 30 14:59:42.279378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.279223 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:42.279378 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.279240 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" event={"ID":"3571d646-2915-4c39-8db1-e35152a7027d","Type":"ContainerStarted","Data":"05593a128dbd6e4663ecc3f797d95d30491ef05cf741900c9d8d45ac334e9d0f"} Oct 30 14:59:42.324940 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.324893 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" podStartSLOduration=1.324879815 podStartE2EDuration="1.324879815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:42.324823237 +0000 UTC m=+686.283864643" watchObservedRunningTime="2026-10-30 14:59:42.324879815 +0000 UTC m=+686.283921219" Oct 30 14:59:42.381634 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:42.381554 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" podStartSLOduration=1.381535687 podStartE2EDuration="1.381535687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:42.378787206 +0000 UTC m=+686.337828611" watchObservedRunningTime="2026-10-30 14:59:42.381535687 +0000 UTC m=+686.340577092" Oct 30 14:59:44.205309 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:44.204586 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 14:59:44.205309 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:44.204649 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 14:59:44.205309 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:44.204743 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 14:59:46.219564 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:46.219534 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 14:59:46.220442 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:46.220424 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 14:59:47.224747 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:47.224721 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 14:59:50.245117 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.245083 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 14:59:50.245543 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.245173 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 14:59:50.584212 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.584134 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b"] Oct 30 14:59:50.591490 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.591457 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b"] Oct 30 14:59:50.591622 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.591561 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.609888 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.609860 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 14:59:50.610109 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.609882 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 14:59:50.610288 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.610003 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 14:59:50.616874 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.616853 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-4b703c-serving-cert\"" Oct 30 14:59:50.617079 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.617064 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4jtfs\"" Oct 30 14:59:50.710883 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.710846 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.711045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.710897 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.711045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.710985 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.711045 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.711029 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkq2\" (UniqueName: \"kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.811661 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.811626 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.811860 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.811685 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.811860 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.811712 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.812107 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.812080 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkq2\" (UniqueName: \"kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.812444 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.812408 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.814159 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.814140 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.814722 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.814699 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.835935 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.835882 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkq2\" (UniqueName: \"kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2\") pod \"maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:50.909985 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:50.909954 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:51.231781 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:51.231756 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b"] Oct 30 14:59:51.233424 ip-10-0-139-204 kubenswrapper[6554]: W1030 14:59:51.233400 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30837e40_0881_430e_a9bb_4260ea32b5f6.slice/crio-1f670801d8b7a3bebe2435b1202e7179f369af7f60e89721354b65d2373f2324 WatchSource:0}: Error finding container 1f670801d8b7a3bebe2435b1202e7179f369af7f60e89721354b65d2373f2324: Status 404 returned error can't find the container with id 1f670801d8b7a3bebe2435b1202e7179f369af7f60e89721354b65d2373f2324 Oct 30 14:59:51.368515 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:51.368479 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" event={"ID":"30837e40-0881-430e-a9bb-4260ea32b5f6","Type":"ContainerStarted","Data":"1f670801d8b7a3bebe2435b1202e7179f369af7f60e89721354b65d2373f2324"} Oct 30 14:59:52.377301 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:52.377210 6554 generic.go:356] "Generic (PLEG): container finished" podID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerID="f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c" exitCode=0 Oct 30 14:59:52.377877 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:52.377392 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" event={"ID":"b85a8334-2fe2-4991-86b4-d4961aefb039","Type":"ContainerDied","Data":"f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c"} Oct 30 14:59:52.377877 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:52.377806 6554 scope.go:122] "RemoveContainer" containerID="f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c" Oct 30 14:59:52.379486 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:52.379445 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:52.379608 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:52.379535 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:52.379662 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:52.379613 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:53.288927 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:53.288899 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 14:59:53.397237 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:53.397196 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" event={"ID":"b85a8334-2fe2-4991-86b4-d4961aefb039","Type":"ContainerStarted","Data":"cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8"} Oct 30 14:59:54.014023 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.013984 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:54.015449 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.015424 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:59:54.015584 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.015511 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:59:54.298416 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.296279 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 14:59:54.415571 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.415532 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" event={"ID":"30837e40-0881-430e-a9bb-4260ea32b5f6","Type":"ContainerStarted","Data":"01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d"} Oct 30 14:59:54.416248 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.416228 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 14:59:54.479890 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:54.479834 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" podStartSLOduration=1.508298909 podStartE2EDuration="4.479814317s" totalImagesPullingTime="2.971515408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:59:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:59:54.468116952 +0000 UTC m=+698.427158361" watchObservedRunningTime="2026-10-30 14:59:54.479814317 +0000 UTC m=+698.438855723" Oct 30 14:59:55.014096 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:55.014054 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:59:55.014308 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:55.014136 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:59:56.014002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.013932 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:59:56.014002 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.013979 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:59:56.138739 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.138685 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:56.139985 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.139947 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:59:56.140118 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.140002 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:59:56.427366 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.427314 6554 generic.go:356] "Generic (PLEG): container finished" podID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerID="1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63" exitCode=0 Oct 30 14:59:56.427540 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.427390 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" event={"ID":"b980b704-8bd8-46ce-b9ee-de7ddc035ec5","Type":"ContainerDied","Data":"1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63"} Oct 30 14:59:56.428210 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.427915 6554 scope.go:122] "RemoveContainer" containerID="1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63" Oct 30 14:59:56.428997 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.428969 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:56.429044 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.429034 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:56.429084 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:56.429073 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 14:59:57.014749 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:57.014716 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:59:57.015149 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:57.014783 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:59:57.437443 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:57.437407 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" event={"ID":"b980b704-8bd8-46ce-b9ee-de7ddc035ec5","Type":"ContainerStarted","Data":"aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f"} Oct 30 14:59:57.667178 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:57.666742 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 14:59:57.667178 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:57.666872 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:59:57.667178 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:57.666918 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:59:58.016812 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:58.016774 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 14:59:58.017830 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:58.017417 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 14:59:58.679373 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:58.674872 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:59:58.679373 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:58.674939 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 14:59:59.150093 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:59.149458 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:59.151138 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:59.151120 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 14:59:59.666374 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:59.665018 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 14:59:59.666374 ip-10-0-139-204 kubenswrapper[6554]: I1030 14:59:59.665087 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 15:00:00.184373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.183545 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd"] Oct 30 15:00:00.235820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.235279 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk"] Oct 30 15:00:00.235820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.235426 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.245463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.245438 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx"] Oct 30 15:00:00.245608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.245545 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.273592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.273563 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd"] Oct 30 15:00:00.276429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.273828 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7"] Oct 30 15:00:00.302065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.302038 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:00:00.305663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.278107 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.326563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.325543 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.326563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.325596 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t48b\" (UniqueName: \"kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.326563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.325653 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktw46\" (UniqueName: \"kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46\") pod \"maas-api-key-cleanup-29889540-5rgbd\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.326563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.325732 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889540-5rgbd\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.326563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.325770 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdx6\" (UniqueName: \"kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.326563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.325816 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.329096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.329058 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk"] Oct 30 15:00:00.329213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.329127 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx"] Oct 30 15:00:00.329213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.329141 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.329213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.329162 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7"] Oct 30 15:00:00.426820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.426783 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtz4l\" (UniqueName: \"kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.426991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.426849 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.426991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.426892 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t48b\" (UniqueName: \"kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.426991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.426913 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.426991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.426938 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktw46\" (UniqueName: \"kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46\") pod \"maas-api-key-cleanup-29889540-5rgbd\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.427222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.427016 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889540-5rgbd\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.427222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.427063 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdx6\" (UniqueName: \"kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.427222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.427106 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.428094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.428069 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.428173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.428069 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889540-5rgbd\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.428726 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.428704 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.474416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.472021 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktw46\" (UniqueName: \"kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46\") pod \"maas-api-key-cleanup-29889540-5rgbd\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.474416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.472613 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t48b\" (UniqueName: \"kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b\") pod \"maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.513182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.513149 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdx6\" (UniqueName: \"kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6\") pod \"maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.527638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.527610 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtz4l\" (UniqueName: \"kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.527758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.527655 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.528312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.528293 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.578485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.578456 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtz4l\" (UniqueName: \"kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:00.610478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.608723 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:00.632025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.631999 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:00.647229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.647200 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:00.692375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.676752 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 15:00:00.692375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.676807 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 15:00:00.696397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:00.693536 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:01.214701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.214283 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd"] Oct 30 15:00:01.218048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.216934 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx"] Oct 30 15:00:01.289426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.289055 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk"] Oct 30 15:00:01.403398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.401451 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7"] Oct 30 15:00:01.410023 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:00:01.409992 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e178aaa_fa2d_4c14_800b_c12d19e57bf0.slice/crio-c6c91325643b5c58c438e483bcd4ef8a87bb0be613a00d78f12775150bfbecb8 WatchSource:0}: Error finding container c6c91325643b5c58c438e483bcd4ef8a87bb0be613a00d78f12775150bfbecb8: Status 404 returned error can't find the container with id c6c91325643b5c58c438e483bcd4ef8a87bb0be613a00d78f12775150bfbecb8 Oct 30 15:00:01.433917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.433890 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 15:00:01.491767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.491677 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerStarted","Data":"c6c91325643b5c58c438e483bcd4ef8a87bb0be613a00d78f12775150bfbecb8"} Oct 30 15:00:01.503432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.493310 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" event={"ID":"721efac0-539d-49ee-a5f7-267e118ef3d3","Type":"ContainerStarted","Data":"14e08a73e8c9410c095195882fe8e4d807d6ebd8ba1b06c8fca671d1eded8312"} Oct 30 15:00:01.505136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.505064 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerStarted","Data":"de20623778c758862d6c38a657a037667c9cd80a365ac765596a5f4f4486665d"} Oct 30 15:00:01.513171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.512493 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerStarted","Data":"998b87b5a7808d8c5be6e30db7e5f8dddc75074a1e46b02c1aa15a02a5935189"} Oct 30 15:00:01.688504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.688463 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 15:00:01.688796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:01.688764 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 15:00:02.673438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:02.673404 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Oct 30 15:00:02.674043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:02.673993 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Oct 30 15:00:03.670622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:03.670592 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 15:00:03.671412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:03.671310 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 15:00:03.675379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:03.672428 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 15:00:06.548370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.548221 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerStarted","Data":"4890e2841321c2564271c50a44e7f1a5d58053fc388532b64c84f6e0dd13559b"} Oct 30 15:00:06.550018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.549992 6554 generic.go:356] "Generic (PLEG): container finished" podID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerID="52cfaad6f6351e412501d0fb54216bb5f99bd8ef8e0c1e7d73223dbe2174eae7" exitCode=60 Oct 30 15:00:06.550148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.550065 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" event={"ID":"721efac0-539d-49ee-a5f7-267e118ef3d3","Type":"ContainerDied","Data":"52cfaad6f6351e412501d0fb54216bb5f99bd8ef8e0c1e7d73223dbe2174eae7"} Oct 30 15:00:06.550332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.550313 6554 scope.go:122] "RemoveContainer" containerID="52cfaad6f6351e412501d0fb54216bb5f99bd8ef8e0c1e7d73223dbe2174eae7" Oct 30 15:00:06.552038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.552011 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerStarted","Data":"ceeb52759aa87ffdd3de33fb9c88afc4795e887df2e1e6f4a72437ec232e5631"} Oct 30 15:00:06.555871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.555389 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerStarted","Data":"5cfc96cb2a1cd30e8ec52e427f59687d08662654d500d5b1b8c965d95c4adaed"} Oct 30 15:00:06.557738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.557710 6554 generic.go:356] "Generic (PLEG): container finished" podID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerID="4447ccfff210a25a263753a91d4f737cd118225ba604a880ed746bf9b5c6c19d" exitCode=0 Oct 30 15:00:06.557830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.557772 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" event={"ID":"1aab4ca9-aa15-4ea5-9919-6585d9bf6718","Type":"ContainerDied","Data":"4447ccfff210a25a263753a91d4f737cd118225ba604a880ed746bf9b5c6c19d"} Oct 30 15:00:06.558118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.558102 6554 scope.go:122] "RemoveContainer" containerID="4447ccfff210a25a263753a91d4f737cd118225ba604a880ed746bf9b5c6c19d" Oct 30 15:00:06.563648 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.563615 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:06.563721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.563707 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:06.563794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.563777 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:06.597569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.597393 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" podStartSLOduration=2.216869174 podStartE2EDuration="6.597377158s" totalImagesPullingTime="4.380507984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:06.595688981 +0000 UTC m=+710.554730387" watchObservedRunningTime="2026-10-30 15:00:06.597377158 +0000 UTC m=+710.556418565" Oct 30 15:00:06.686383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.684215 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" podStartSLOduration=2.126956704 podStartE2EDuration="6.684193893s" totalImagesPullingTime="4.557237189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:06.683778483 +0000 UTC m=+710.642819889" watchObservedRunningTime="2026-10-30 15:00:06.684193893 +0000 UTC m=+710.643235345" Oct 30 15:00:06.737368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:06.736942 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" podStartSLOduration=2.101070225 podStartE2EDuration="6.736919597s" totalImagesPullingTime="4.635849372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:06.734419437 +0000 UTC m=+710.693460838" watchObservedRunningTime="2026-10-30 15:00:06.736919597 +0000 UTC m=+710.695961015" Oct 30 15:00:07.580886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.580853 6554 generic.go:356] "Generic (PLEG): container finished" podID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerID="5cfc96cb2a1cd30e8ec52e427f59687d08662654d500d5b1b8c965d95c4adaed" exitCode=60 Oct 30 15:00:07.581330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.580909 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerDied","Data":"5cfc96cb2a1cd30e8ec52e427f59687d08662654d500d5b1b8c965d95c4adaed"} Oct 30 15:00:07.581330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.581277 6554 scope.go:122] "RemoveContainer" containerID="5cfc96cb2a1cd30e8ec52e427f59687d08662654d500d5b1b8c965d95c4adaed" Oct 30 15:00:07.583619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.583592 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" event={"ID":"1aab4ca9-aa15-4ea5-9919-6585d9bf6718","Type":"ContainerStarted","Data":"1ad0475590c1dfc000f4ee3ae7a282b9e46e0cad76a4873d190f9e6e6e8c1a8e"} Oct 30 15:00:07.585175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.585156 6554 generic.go:356] "Generic (PLEG): container finished" podID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerID="4890e2841321c2564271c50a44e7f1a5d58053fc388532b64c84f6e0dd13559b" exitCode=60 Oct 30 15:00:07.585298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.585178 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerDied","Data":"4890e2841321c2564271c50a44e7f1a5d58053fc388532b64c84f6e0dd13559b"} Oct 30 15:00:07.585476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.585460 6554 scope.go:122] "RemoveContainer" containerID="4890e2841321c2564271c50a44e7f1a5d58053fc388532b64c84f6e0dd13559b" Oct 30 15:00:07.587204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.587185 6554 generic.go:356] "Generic (PLEG): container finished" podID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerID="55a37f4d30f42664da65c20b19cc7f640c5bbb0d68c582cca98b458534a23b80" exitCode=60 Oct 30 15:00:07.587307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.587258 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" event={"ID":"721efac0-539d-49ee-a5f7-267e118ef3d3","Type":"ContainerDied","Data":"55a37f4d30f42664da65c20b19cc7f640c5bbb0d68c582cca98b458534a23b80"} Oct 30 15:00:07.587307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.587301 6554 scope.go:122] "RemoveContainer" containerID="52cfaad6f6351e412501d0fb54216bb5f99bd8ef8e0c1e7d73223dbe2174eae7" Oct 30 15:00:07.587539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.587521 6554 scope.go:122] "RemoveContainer" containerID="55a37f4d30f42664da65c20b19cc7f640c5bbb0d68c582cca98b458534a23b80" Oct 30 15:00:07.587736 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:07.587716 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk_odh-ai-gateway-infra(721efac0-539d-49ee-a5f7-267e118ef3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" Oct 30 15:00:07.588849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.588828 6554 generic.go:356] "Generic (PLEG): container finished" podID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerID="ceeb52759aa87ffdd3de33fb9c88afc4795e887df2e1e6f4a72437ec232e5631" exitCode=60 Oct 30 15:00:07.588949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.588884 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerDied","Data":"ceeb52759aa87ffdd3de33fb9c88afc4795e887df2e1e6f4a72437ec232e5631"} Oct 30 15:00:07.589031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:07.589018 6554 scope.go:122] "RemoveContainer" containerID="ceeb52759aa87ffdd3de33fb9c88afc4795e887df2e1e6f4a72437ec232e5631" Oct 30 15:00:08.175374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.165652 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 15:00:08.175374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.173450 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:00:08.175374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.173500 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 15:00:08.596546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.596520 6554 generic.go:356] "Generic (PLEG): container finished" podID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerID="d365c1590acfdbeefb795bb384c82e36bcdaf461cd4c6c3ee87c19a071f85ddb" exitCode=60 Oct 30 15:00:08.596917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.596604 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerDied","Data":"d365c1590acfdbeefb795bb384c82e36bcdaf461cd4c6c3ee87c19a071f85ddb"} Oct 30 15:00:08.596917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.596677 6554 scope.go:122] "RemoveContainer" containerID="4890e2841321c2564271c50a44e7f1a5d58053fc388532b64c84f6e0dd13559b" Oct 30 15:00:08.597038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.596963 6554 scope.go:122] "RemoveContainer" containerID="d365c1590acfdbeefb795bb384c82e36bcdaf461cd4c6c3ee87c19a071f85ddb" Oct 30 15:00:08.597260 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:08.597235 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7_odh-ai-gateway-infra(2e178aaa-fa2d-4c14-800b-c12d19e57bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" Oct 30 15:00:08.598878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.598861 6554 scope.go:122] "RemoveContainer" containerID="55a37f4d30f42664da65c20b19cc7f640c5bbb0d68c582cca98b458534a23b80" Oct 30 15:00:08.599103 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:08.599083 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk_odh-ai-gateway-infra(721efac0-539d-49ee-a5f7-267e118ef3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" Oct 30 15:00:08.600605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.600583 6554 generic.go:356] "Generic (PLEG): container finished" podID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerID="ee1ae8e27f1f75b4263e55062f1169723d42259052acc2f1380a901c72695d49" exitCode=60 Oct 30 15:00:08.600734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.600665 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerDied","Data":"ee1ae8e27f1f75b4263e55062f1169723d42259052acc2f1380a901c72695d49"} Oct 30 15:00:08.600856 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.600844 6554 scope.go:122] "RemoveContainer" containerID="ee1ae8e27f1f75b4263e55062f1169723d42259052acc2f1380a901c72695d49" Oct 30 15:00:08.601078 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:08.601053 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889540-5rgbd_odh-ai-gateway-infra(351c58c9-f004-4e04-8186-d29fa8651bc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" Oct 30 15:00:08.602853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.602831 6554 generic.go:356] "Generic (PLEG): container finished" podID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerID="00e319535ffcb9695b64c934e53d278d383d35441c7df93bdc6c7e5973411ed4" exitCode=60 Oct 30 15:00:08.602925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.602887 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerDied","Data":"00e319535ffcb9695b64c934e53d278d383d35441c7df93bdc6c7e5973411ed4"} Oct 30 15:00:08.603126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.603110 6554 scope.go:122] "RemoveContainer" containerID="00e319535ffcb9695b64c934e53d278d383d35441c7df93bdc6c7e5973411ed4" Oct 30 15:00:08.603387 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:08.603328 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx_odh-ai-gateway-infra(b160dbe7-a3f3-4453-9113-ae5c49cdbf0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" Oct 30 15:00:08.784172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.784149 6554 scope.go:122] "RemoveContainer" containerID="ceeb52759aa87ffdd3de33fb9c88afc4795e887df2e1e6f4a72437ec232e5631" Oct 30 15:00:08.798529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:08.798509 6554 scope.go:122] "RemoveContainer" containerID="5cfc96cb2a1cd30e8ec52e427f59687d08662654d500d5b1b8c965d95c4adaed" Oct 30 15:00:09.175374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:09.166672 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:00:09.175374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:09.166730 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 15:00:09.206266 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:09.206224 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:00:09.610239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:09.610173 6554 scope.go:122] "RemoveContainer" containerID="00e319535ffcb9695b64c934e53d278d383d35441c7df93bdc6c7e5973411ed4" Oct 30 15:00:09.610663 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:09.610464 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx_odh-ai-gateway-infra(b160dbe7-a3f3-4453-9113-ae5c49cdbf0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" Oct 30 15:00:09.611800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:09.611779 6554 scope.go:122] "RemoveContainer" containerID="d365c1590acfdbeefb795bb384c82e36bcdaf461cd4c6c3ee87c19a071f85ddb" Oct 30 15:00:09.611999 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:09.611980 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7_odh-ai-gateway-infra(2e178aaa-fa2d-4c14-800b-c12d19e57bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" Oct 30 15:00:09.613311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:09.613295 6554 scope.go:122] "RemoveContainer" containerID="ee1ae8e27f1f75b4263e55062f1169723d42259052acc2f1380a901c72695d49" Oct 30 15:00:09.613493 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:09.613475 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889540-5rgbd_odh-ai-gateway-infra(351c58c9-f004-4e04-8186-d29fa8651bc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" Oct 30 15:00:10.181399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:10.178498 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:00:10.181399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:10.178611 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 15:00:11.139600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:11.139572 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 15:00:11.176371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:11.175464 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:00:11.176371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:11.175571 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 15:00:12.178373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:12.168708 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 15:00:12.178373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:12.168798 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 15:00:13.165008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:13.164978 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 15:00:13.166209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:13.166185 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 15:00:19.147595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:19.147563 6554 scope.go:122] "RemoveContainer" containerID="ee1ae8e27f1f75b4263e55062f1169723d42259052acc2f1380a901c72695d49" Oct 30 15:00:19.672743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:19.672714 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerStarted","Data":"99ed77d3cd4c3edbb60480fd2d9006c34eec50cc9fb5b5a1d569489c6ebe14eb"} Oct 30 15:00:20.679941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:20.679911 6554 generic.go:356] "Generic (PLEG): container finished" podID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerID="99ed77d3cd4c3edbb60480fd2d9006c34eec50cc9fb5b5a1d569489c6ebe14eb" exitCode=60 Oct 30 15:00:20.680393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:20.679983 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerDied","Data":"99ed77d3cd4c3edbb60480fd2d9006c34eec50cc9fb5b5a1d569489c6ebe14eb"} Oct 30 15:00:20.680393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:20.680041 6554 scope.go:122] "RemoveContainer" containerID="ee1ae8e27f1f75b4263e55062f1169723d42259052acc2f1380a901c72695d49" Oct 30 15:00:20.680393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:20.680307 6554 scope.go:122] "RemoveContainer" containerID="99ed77d3cd4c3edbb60480fd2d9006c34eec50cc9fb5b5a1d569489c6ebe14eb" Oct 30 15:00:20.680573 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:20.680543 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889540-5rgbd_odh-ai-gateway-infra(351c58c9-f004-4e04-8186-d29fa8651bc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" Oct 30 15:00:20.736140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:20.736114 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd"] Oct 30 15:00:21.147458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:21.147436 6554 scope.go:122] "RemoveContainer" containerID="00e319535ffcb9695b64c934e53d278d383d35441c7df93bdc6c7e5973411ed4" Oct 30 15:00:21.690456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:21.690415 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerStarted","Data":"0b07c227a48eee0f4011d6d3b76feb150711b9da944a05059104896a10df3e47"} Oct 30 15:00:22.056712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.056683 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:22.142047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142011 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh"] Oct 30 15:00:22.142549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142529 6554 cpu_manager.go:414] "containerMap: removing container" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:22.142549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142548 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:22.142715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142590 6554 cpu_manager.go:414] "containerMap: removing container" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:22.142715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142595 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:22.142715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142681 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:22.142715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.142690 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:22.145904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.145879 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.146902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.146880 6554 scope.go:122] "RemoveContainer" containerID="55a37f4d30f42664da65c20b19cc7f640c5bbb0d68c582cca98b458534a23b80" Oct 30 15:00:22.147500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.147478 6554 scope.go:122] "RemoveContainer" containerID="d365c1590acfdbeefb795bb384c82e36bcdaf461cd4c6c3ee87c19a071f85ddb" Oct 30 15:00:22.170844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.170637 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca\") pod \"351c58c9-f004-4e04-8186-d29fa8651bc4\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " Oct 30 15:00:22.170844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.170680 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"openshift-service-ca.crt\"" Oct 30 15:00:22.170844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.170813 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46\" (UniqueName: \"kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46\") pod \"351c58c9-f004-4e04-8186-d29fa8651bc4\" (UID: \"351c58c9-f004-4e04-8186-d29fa8651bc4\") " Oct 30 15:00:22.174287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.173646 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca" pod "351c58c9-f004-4e04-8186-d29fa8651bc4" (UID: "351c58c9-f004-4e04-8186-d29fa8651bc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:22.174287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.173807 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"kube-root-ca.crt\"" Oct 30 15:00:22.174287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.173850 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"default-dockercfg-d6t6d\"" Oct 30 15:00:22.174287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.173959 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"simulator-w6-2c8d89-kserve-self-signed-certs\"" Oct 30 15:00:22.175159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.175121 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46" pod "351c58c9-f004-4e04-8186-d29fa8651bc4" (UID: "351c58c9-f004-4e04-8186-d29fa8651bc4"). InnerVolumeSpecName "kube-api-access-ktw46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:22.177605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.177581 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh"] Oct 30 15:00:22.272282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272262 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.272403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272311 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.272526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272505 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlpm\" (UniqueName: \"kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.272619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272602 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.272657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272638 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.272774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272758 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.272843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272831 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktw46\" (UniqueName: \"kubernetes.io/projected/351c58c9-f004-4e04-8186-d29fa8651bc4-kube-api-access-ktw46\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:22.272879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.272848 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/351c58c9-f004-4e04-8186-d29fa8651bc4-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.373916 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.373998 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374080 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374131 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374175 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlpm\" (UniqueName: \"kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374210 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.374463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374333 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.378370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374521 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.378370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.374741 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.380493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.380468 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.454469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.454431 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.454629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.454471 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlpm\" (UniqueName: \"kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm\") pod \"simulator-w6-2c8d89-kserve-5f6d56469-9flkh\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.483303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.483279 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:22.697160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.697132 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerStarted","Data":"ae60814add4ef7db55f7879cf55a627813f5502a0cf170cc3ca290857f83ffcd"} Oct 30 15:00:22.699078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.699054 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" event={"ID":"721efac0-539d-49ee-a5f7-267e118ef3d3","Type":"ContainerStarted","Data":"e302e0022ad712f6f0e865fb2ddaed1f6b95dbadf5d93a163cad87d96dfce33c"} Oct 30 15:00:22.700225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.700209 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" Oct 30 15:00:22.700334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.700258 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd" event={"ID":"351c58c9-f004-4e04-8186-d29fa8651bc4","Type":"ContainerDied","Data":"de20623778c758862d6c38a657a037667c9cd80a365ac765596a5f4f4486665d"} Oct 30 15:00:22.700334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.700307 6554 scope.go:122] "RemoveContainer" containerID="99ed77d3cd4c3edbb60480fd2d9006c34eec50cc9fb5b5a1d569489c6ebe14eb" Oct 30 15:00:22.702231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.702206 6554 generic.go:356] "Generic (PLEG): container finished" podID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerID="0b07c227a48eee0f4011d6d3b76feb150711b9da944a05059104896a10df3e47" exitCode=60 Oct 30 15:00:22.702367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.702276 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerDied","Data":"0b07c227a48eee0f4011d6d3b76feb150711b9da944a05059104896a10df3e47"} Oct 30 15:00:22.702651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.702633 6554 scope.go:122] "RemoveContainer" containerID="0b07c227a48eee0f4011d6d3b76feb150711b9da944a05059104896a10df3e47" Oct 30 15:00:22.702840 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:22.702823 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx_odh-ai-gateway-infra(b160dbe7-a3f3-4453-9113-ae5c49cdbf0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" Oct 30 15:00:22.708781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.708761 6554 scope.go:122] "RemoveContainer" containerID="00e319535ffcb9695b64c934e53d278d383d35441c7df93bdc6c7e5973411ed4" Oct 30 15:00:22.782759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.782739 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh"] Oct 30 15:00:22.784590 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:00:22.784566 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4617f1de_b33b_4240_b155_972e4e8cf425.slice/crio-77f7a995f0302b34202ebaff9d3f4f30db09dcbbaf5a01755a0d043a113c07fa WatchSource:0}: Error finding container 77f7a995f0302b34202ebaff9d3f4f30db09dcbbaf5a01755a0d043a113c07fa: Status 404 returned error can't find the container with id 77f7a995f0302b34202ebaff9d3f4f30db09dcbbaf5a01755a0d043a113c07fa Oct 30 15:00:22.805072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.805046 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx"] Oct 30 15:00:22.832649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.832611 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" podStartSLOduration=18.745651987 podStartE2EDuration="22.832599338s" totalImagesPullingTime="4.086947351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:22.831648372 +0000 UTC m=+726.790689777" watchObservedRunningTime="2026-10-30 15:00:22.832599338 +0000 UTC m=+726.791640743" Oct 30 15:00:22.877787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.877760 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd"] Oct 30 15:00:22.884718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:22.884695 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889540-5rgbd"] Oct 30 15:00:23.221733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.221681 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs"] Oct 30 15:00:23.222175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.222161 6554 cpu_manager.go:414] "containerMap: removing container" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:23.222216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.222176 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:23.222294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.222285 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" containerName="cleanup" Oct 30 15:00:23.225531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.225514 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.247673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.247563 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e64e05-openshift-default-dockercfg-s2txv\"" Oct 30 15:00:23.249874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.249853 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs"] Oct 30 15:00:23.285056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285030 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285145 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285208 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285256 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285373 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285417 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285475 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285577 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sv2k\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.285650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.285637 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.386846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.386823 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.386956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.386873 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.386956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.386900 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.386952 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.386997 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387028 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387072 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387141 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sv2k\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387187 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387203 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387248 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387386 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387518 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.387838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.387815 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.389249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.389226 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.389466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.389451 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.413186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.413158 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.413314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.413221 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sv2k\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k\") pod \"e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.539111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.539048 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:23.709003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.708974 6554 generic.go:356] "Generic (PLEG): container finished" podID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerID="e302e0022ad712f6f0e865fb2ddaed1f6b95dbadf5d93a163cad87d96dfce33c" exitCode=60 Oct 30 15:00:23.709410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.709047 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" event={"ID":"721efac0-539d-49ee-a5f7-267e118ef3d3","Type":"ContainerDied","Data":"e302e0022ad712f6f0e865fb2ddaed1f6b95dbadf5d93a163cad87d96dfce33c"} Oct 30 15:00:23.709410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.709099 6554 scope.go:122] "RemoveContainer" containerID="55a37f4d30f42664da65c20b19cc7f640c5bbb0d68c582cca98b458534a23b80" Oct 30 15:00:23.709535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.709433 6554 scope.go:122] "RemoveContainer" containerID="e302e0022ad712f6f0e865fb2ddaed1f6b95dbadf5d93a163cad87d96dfce33c" Oct 30 15:00:23.709711 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:23.709690 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk_odh-ai-gateway-infra(721efac0-539d-49ee-a5f7-267e118ef3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" Oct 30 15:00:23.711940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.711919 6554 generic.go:356] "Generic (PLEG): container finished" podID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerID="ae60814add4ef7db55f7879cf55a627813f5502a0cf170cc3ca290857f83ffcd" exitCode=60 Oct 30 15:00:23.712026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.711979 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerDied","Data":"ae60814add4ef7db55f7879cf55a627813f5502a0cf170cc3ca290857f83ffcd"} Oct 30 15:00:23.712312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.712293 6554 scope.go:122] "RemoveContainer" containerID="ae60814add4ef7db55f7879cf55a627813f5502a0cf170cc3ca290857f83ffcd" Oct 30 15:00:23.712540 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:23.712518 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7_odh-ai-gateway-infra(2e178aaa-fa2d-4c14-800b-c12d19e57bf0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" Oct 30 15:00:23.715134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.715113 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" event={"ID":"4617f1de-b33b-4240-b155-972e4e8cf425","Type":"ContainerStarted","Data":"1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4"} Oct 30 15:00:23.715239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.715142 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" event={"ID":"4617f1de-b33b-4240-b155-972e4e8cf425","Type":"ContainerStarted","Data":"77f7a995f0302b34202ebaff9d3f4f30db09dcbbaf5a01755a0d043a113c07fa"} Oct 30 15:00:23.722378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.722343 6554 scope.go:122] "RemoveContainer" containerID="d365c1590acfdbeefb795bb384c82e36bcdaf461cd4c6c3ee87c19a071f85ddb" Oct 30 15:00:23.781516 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.781483 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7"] Oct 30 15:00:23.836313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.836278 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:23.857601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.857570 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk"] Oct 30 15:00:23.888963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.886442 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs"] Oct 30 15:00:23.888963 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:00:23.888568 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e3f5a2c_88f7_49ef_90fd_f37dbae240ad.slice/crio-efbe226ce684a32384526b62717da8c55138e9a4ba52d2c0888880798733ddae WatchSource:0}: Error finding container efbe226ce684a32384526b62717da8c55138e9a4ba52d2c0888880798733ddae: Status 404 returned error can't find the container with id efbe226ce684a32384526b62717da8c55138e9a4ba52d2c0888880798733ddae Oct 30 15:00:23.891797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.891770 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:23.891870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.891825 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:23.891870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.891853 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:23.892476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.892374 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca\") pod \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " Oct 30 15:00:23.892556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.892523 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b\" (UniqueName: \"kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b\") pod \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\" (UID: \"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a\") " Oct 30 15:00:23.892785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.892746 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca" pod "b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" (UID: "b160dbe7-a3f3-4453-9113-ae5c49cdbf0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:23.892981 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.892920 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:23.894928 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.894903 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b" pod "b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" (UID: "b160dbe7-a3f3-4453-9113-ae5c49cdbf0a"). InnerVolumeSpecName "kube-api-access-2t48b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:23.917214 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.917170 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podStartSLOduration=1.91715859 podStartE2EDuration="1.91715859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:23.914872884 +0000 UTC m=+727.873914290" watchObservedRunningTime="2026-10-30 15:00:23.91715859 +0000 UTC m=+727.876199992" Oct 30 15:00:23.994495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:23.994472 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t48b\" (UniqueName: \"kubernetes.io/projected/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a-kube-api-access-2t48b\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:24.154060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.154028 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351c58c9-f004-4e04-8186-d29fa8651bc4" path="/var/lib/kubelet/pods/351c58c9-f004-4e04-8186-d29fa8651bc4/volumes" Oct 30 15:00:24.484535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.484447 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:24.723389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.723335 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" event={"ID":"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad","Type":"ContainerStarted","Data":"4c8e2cb350c535b4ec993c8a6854b8af26c8cca58bea5ec1af452a5cb02c2fb0"} Oct 30 15:00:24.723770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.723424 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" event={"ID":"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad","Type":"ContainerStarted","Data":"efbe226ce684a32384526b62717da8c55138e9a4ba52d2c0888880798733ddae"} Oct 30 15:00:24.727561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.727531 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" event={"ID":"b160dbe7-a3f3-4453-9113-ae5c49cdbf0a","Type":"ContainerDied","Data":"998b87b5a7808d8c5be6e30db7e5f8dddc75074a1e46b02c1aa15a02a5935189"} Oct 30 15:00:24.727561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.727556 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx" Oct 30 15:00:24.727757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.727594 6554 scope.go:122] "RemoveContainer" containerID="0b07c227a48eee0f4011d6d3b76feb150711b9da944a05059104896a10df3e47" Oct 30 15:00:24.827608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.827558 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podStartSLOduration=1.827543854 podStartE2EDuration="1.827543854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:24.825666476 +0000 UTC m=+728.784707886" watchObservedRunningTime="2026-10-30 15:00:24.827543854 +0000 UTC m=+728.786585258" Oct 30 15:00:24.920202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.920177 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:24.926102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.925931 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx"] Oct 30 15:00:24.928964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.928942 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-2c8d89-29889540-ffrkx"] Oct 30 15:00:24.947520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:24.947502 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:25.007667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.007578 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6\" (UniqueName: \"kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6\") pod \"721efac0-539d-49ee-a5f7-267e118ef3d3\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " Oct 30 15:00:25.007838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.007673 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca\") pod \"721efac0-539d-49ee-a5f7-267e118ef3d3\" (UID: \"721efac0-539d-49ee-a5f7-267e118ef3d3\") " Oct 30 15:00:25.008046 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.008023 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca" pod "721efac0-539d-49ee-a5f7-267e118ef3d3" (UID: "721efac0-539d-49ee-a5f7-267e118ef3d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:25.008186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.008167 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/721efac0-539d-49ee-a5f7-267e118ef3d3-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:25.009656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.009635 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6" pod "721efac0-539d-49ee-a5f7-267e118ef3d3" (UID: "721efac0-539d-49ee-a5f7-267e118ef3d3"). InnerVolumeSpecName "kube-api-access-6tdx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:25.108579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.108546 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca\") pod \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " Oct 30 15:00:25.108745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.108662 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l\" (UniqueName: \"kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l\") pod \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\" (UID: \"2e178aaa-fa2d-4c14-800b-c12d19e57bf0\") " Oct 30 15:00:25.108962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.108934 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca" pod "2e178aaa-fa2d-4c14-800b-c12d19e57bf0" (UID: "2e178aaa-fa2d-4c14-800b-c12d19e57bf0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:00:25.109101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.109083 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:25.109157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.109107 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tdx6\" (UniqueName: \"kubernetes.io/projected/721efac0-539d-49ee-a5f7-267e118ef3d3-kube-api-access-6tdx6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:25.110623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.110603 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l" pod "2e178aaa-fa2d-4c14-800b-c12d19e57bf0" (UID: "2e178aaa-fa2d-4c14-800b-c12d19e57bf0"). InnerVolumeSpecName "kube-api-access-mtz4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:00:25.210233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.210189 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtz4l\" (UniqueName: \"kubernetes.io/projected/2e178aaa-fa2d-4c14-800b-c12d19e57bf0-kube-api-access-mtz4l\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:00:25.539424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.539395 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:25.540711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.540681 6554 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 15:00:25.540850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.540728 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 15:00:25.733996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.733972 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" Oct 30 15:00:25.734500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.734013 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk" event={"ID":"721efac0-539d-49ee-a5f7-267e118ef3d3","Type":"ContainerDied","Data":"14e08a73e8c9410c095195882fe8e4d807d6ebd8ba1b06c8fca671d1eded8312"} Oct 30 15:00:25.734500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.734079 6554 scope.go:122] "RemoveContainer" containerID="e302e0022ad712f6f0e865fb2ddaed1f6b95dbadf5d93a163cad87d96dfce33c" Oct 30 15:00:25.735669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.735650 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" Oct 30 15:00:25.735750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.735693 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7" event={"ID":"2e178aaa-fa2d-4c14-800b-c12d19e57bf0","Type":"ContainerDied","Data":"c6c91325643b5c58c438e483bcd4ef8a87bb0be613a00d78f12775150bfbecb8"} Oct 30 15:00:25.744079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.744064 6554 scope.go:122] "RemoveContainer" containerID="ae60814add4ef7db55f7879cf55a627813f5502a0cf170cc3ca290857f83ffcd" Oct 30 15:00:25.819476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.819424 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk"] Oct 30 15:00:25.832665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.832643 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-4f5c30-29889540-hmkkk"] Oct 30 15:00:25.886182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.886153 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7"] Oct 30 15:00:25.888911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:25.888890 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889540-dptl7"] Oct 30 15:00:26.161381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:26.154040 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" path="/var/lib/kubelet/pods/2e178aaa-fa2d-4c14-800b-c12d19e57bf0/volumes" Oct 30 15:00:26.161381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:26.154623 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" path="/var/lib/kubelet/pods/721efac0-539d-49ee-a5f7-267e118ef3d3/volumes" Oct 30 15:00:26.161381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:26.155129 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" path="/var/lib/kubelet/pods/b160dbe7-a3f3-4453-9113-ae5c49cdbf0a/volumes" Oct 30 15:00:26.539968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:26.539894 6554 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 15:00:26.539968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:26.539947 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 15:00:27.539722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:27.539686 6554 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 15:00:27.540084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:27.539735 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 15:00:28.540036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:28.540001 6554 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 15:00:28.540449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:28.540061 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 15:00:29.539695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:29.539663 6554 patch_prober.go:29] interesting pod/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 15:00:29.539879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:29.539720 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 15:00:31.383172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:31.383139 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:31.383580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:31.383469 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:31.384360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:31.384328 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:00:32.483842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.483810 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:32.498099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.498070 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:32.571173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571145 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv"] Oct 30 15:00:32.571608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571591 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.571608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571606 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571615 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571620 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571630 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571635 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571651 6554 cpu_manager.go:414] "containerMap: removing container" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571656 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571678 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571684 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571692 6554 cpu_manager.go:414] "containerMap: removing container" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571696 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571759 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571770 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571776 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.571775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571782 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.572540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571789 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:32.572540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571795 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.572540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571799 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:32.572540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.571805 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:32.575679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.575657 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.593583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.593560 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"kube-root-ca.crt\"" Oct 30 15:00:32.593720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.593647 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"openshift-service-ca.crt\"" Oct 30 15:00:32.594252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.594235 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"default-dockercfg-5jx9k\"" Oct 30 15:00:32.594338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.594266 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"premium-w5-4f5c30-kserve-self-signed-certs\"" Oct 30 15:00:32.595990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.595971 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv"] Oct 30 15:00:32.717809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.717785 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmttb\" (UniqueName: \"kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.717956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.717833 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.717956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.717865 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.717956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.717933 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.718068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.717998 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.718068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.718034 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.798201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.798121 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:00:32.819455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819429 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819471 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819518 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmttb\" (UniqueName: \"kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819564 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819592 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819621 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819847 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819920 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.819982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.819938 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.821913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.821884 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.822238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.822220 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.845200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.845179 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmttb\" (UniqueName: \"kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb\") pod \"premium-w5-4f5c30-kserve-d79d9dddb-h84fv\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:32.891439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:32.891413 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:33.192532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:33.192508 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv"] Oct 30 15:00:33.193801 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:00:33.193779 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6b0c106_9b7e_4f7b_9aea_15bf5272a65a.slice/crio-00076d9ee34a7fbc8496bfe8cefadafc76e42a14f9af614eb4b144e4f07fcf99 WatchSource:0}: Error finding container 00076d9ee34a7fbc8496bfe8cefadafc76e42a14f9af614eb4b144e4f07fcf99: Status 404 returned error can't find the container with id 00076d9ee34a7fbc8496bfe8cefadafc76e42a14f9af614eb4b144e4f07fcf99 Oct 30 15:00:33.796536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:33.796485 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" event={"ID":"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a","Type":"ContainerStarted","Data":"1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c"} Oct 30 15:00:33.797025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:33.796554 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" event={"ID":"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a","Type":"ContainerStarted","Data":"00076d9ee34a7fbc8496bfe8cefadafc76e42a14f9af614eb4b144e4f07fcf99"} Oct 30 15:00:33.842542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:33.842499 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podStartSLOduration=1.84248544 podStartE2EDuration="1.84248544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:33.841018847 +0000 UTC m=+737.800060249" watchObservedRunningTime="2026-10-30 15:00:33.84248544 +0000 UTC m=+737.801526895" Oct 30 15:00:33.892123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:33.892101 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:37.672395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.672345 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml"] Oct 30 15:00:37.673160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673131 6554 cpu_manager.go:414] "containerMap: removing container" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:37.673160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673158 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:37.673332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673217 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:37.673332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673226 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b160dbe7-a3f3-4453-9113-ae5c49cdbf0a" containerName="cleanup" Oct 30 15:00:37.673332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673240 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:37.673332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673247 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e178aaa-fa2d-4c14-800b-c12d19e57bf0" containerName="cleanup" Oct 30 15:00:37.673497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.673371 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="721efac0-539d-49ee-a5f7-267e118ef3d3" containerName="cleanup" Oct 30 15:00:37.679610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.679584 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.697609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.697583 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml"] Oct 30 15:00:37.708278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.708249 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"openshift-service-ca.crt\"" Oct 30 15:00:37.708425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.708337 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"default-dockercfg-2jzcj\"" Oct 30 15:00:37.721364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.711103 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"premium-w0-05d396-kserve-self-signed-certs\"" Oct 30 15:00:37.721364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.711319 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-05d396\"/\"kube-root-ca.crt\"" Oct 30 15:00:37.770337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.770313 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czlp\" (UniqueName: \"kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.770465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.770378 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.770465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.770452 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.770606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.770587 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.770674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.770643 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.770731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.770692 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.871695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.871617 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.871919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.871900 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.872139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.872120 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.872416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.872392 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.872602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.872402 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czlp\" (UniqueName: \"kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.872875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.872673 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.873057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.873038 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.873219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.873203 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.882207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.873740 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.888936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.888896 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.889418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.889370 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.900785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.900733 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czlp\" (UniqueName: \"kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp\") pod \"premium-w0-05d396-kserve-8684968f8f-x5jml\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:37.997373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:37.996505 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:38.302887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:38.302861 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml"] Oct 30 15:00:38.304553 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:00:38.304524 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda62ac25a_76ca_4fee_9999_180c82bb8122.slice/crio-2398b702a7038b98b003982d3863f8aa1dddcd1dd29bdcac6e183123c4a3ed1f WatchSource:0}: Error finding container 2398b702a7038b98b003982d3863f8aa1dddcd1dd29bdcac6e183123c4a3ed1f: Status 404 returned error can't find the container with id 2398b702a7038b98b003982d3863f8aa1dddcd1dd29bdcac6e183123c4a3ed1f Oct 30 15:00:38.830072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:38.830038 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" event={"ID":"a62ac25a-76ca-4fee-9999-180c82bb8122","Type":"ContainerStarted","Data":"a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8"} Oct 30 15:00:38.830477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:38.830085 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" event={"ID":"a62ac25a-76ca-4fee-9999-180c82bb8122","Type":"ContainerStarted","Data":"2398b702a7038b98b003982d3863f8aa1dddcd1dd29bdcac6e183123c4a3ed1f"} Oct 30 15:00:38.879025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:38.878980 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podStartSLOduration=1.878967093 podStartE2EDuration="1.878967093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:38.878557713 +0000 UTC m=+742.837599119" watchObservedRunningTime="2026-10-30 15:00:38.878967093 +0000 UTC m=+742.838008498" Oct 30 15:00:38.996847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:38.996818 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:39.200981 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:00:39.200943 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:00:42.892396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:42.892343 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:42.916295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:42.916249 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:43.869447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:43.869411 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:00:47.997330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:47.997303 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:48.009925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:48.009899 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:48.895456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:48.895429 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:00:53.796244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.796210 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt"] Oct 30 15:00:53.801585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.801564 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.831233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.831210 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-2c8d89-openshift-default-dockercfg-mxn4h\"" Oct 30 15:00:53.834105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.834080 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt"] Oct 30 15:00:53.857633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857591 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857661 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857711 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857736 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857794 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhhl5\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857858 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857892 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.857947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857918 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.858155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.857988 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959188 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959223 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959273 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959301 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959344 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhhl5\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959441 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959478 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959502 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959545 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959705 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959798 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.959911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959873 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.960027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.959989 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.960256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.960234 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.961465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.961446 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.961561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.961523 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.985590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.985562 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhhl5\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:53.985723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:53.985665 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:54.115929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.115867 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:54.434836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.434809 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt"] Oct 30 15:00:54.436036 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:00:54.436010 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154c6b17_aa49_46c2_9cda_7d795d69492f.slice/crio-5ebc7f533b0ccb8d482207c34622e51bfca5d66320c3954594aeed17cd36abdf WatchSource:0}: Error finding container 5ebc7f533b0ccb8d482207c34622e51bfca5d66320c3954594aeed17cd36abdf: Status 404 returned error can't find the container with id 5ebc7f533b0ccb8d482207c34622e51bfca5d66320c3954594aeed17cd36abdf Oct 30 15:00:54.438157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.438128 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:54.438221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.438190 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:54.438261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.438218 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:00:54.917698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.917661 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" event={"ID":"154c6b17-aa49-46c2-9cda-7d795d69492f","Type":"ContainerStarted","Data":"f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e"} Oct 30 15:00:54.918168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.917716 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" event={"ID":"154c6b17-aa49-46c2-9cda-7d795d69492f","Type":"ContainerStarted","Data":"5ebc7f533b0ccb8d482207c34622e51bfca5d66320c3954594aeed17cd36abdf"} Oct 30 15:00:54.986213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:54.986162 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podStartSLOduration=1.9861486350000002 podStartE2EDuration="1.986148635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:00:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:00:54.984263316 +0000 UTC m=+758.943304734" watchObservedRunningTime="2026-10-30 15:00:54.986148635 +0000 UTC m=+758.945190040" Oct 30 15:00:55.116679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:55.116646 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:00:55.118041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:55.118014 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 15:00:55.118172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:55.118061 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 15:00:56.116904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:56.116864 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 15:00:56.117428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:56.116920 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 15:00:57.116744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:57.116705 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 15:00:57.116917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:57.116768 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 15:00:58.116499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:58.116464 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 15:00:58.116904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:58.116513 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 15:00:59.131370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:59.122725 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 15:00:59.131370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:00:59.122782 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 15:01:00.116647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:00.116613 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 15:01:00.116818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:00.116664 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 15:01:01.120952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:01.120921 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:01:01.121379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:01.121203 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:01:01.122041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:01.122021 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:01:08.493874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.493832 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9"] Oct 30 15:01:08.497785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.497760 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.517304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.517257 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-4f5c30-openshift-default-dockercfg-pvckr\"" Oct 30 15:01:08.533599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.533564 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9"] Oct 30 15:01:08.642884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.642857 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.642894 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.642916 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.643033 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.643088 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.643174 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.643213 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.643244 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qpj\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.643294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.643284 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744555 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744586 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744608 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744652 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744693 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744721 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744754 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744777 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qpj\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.744895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744817 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.745194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.744973 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.745194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.745045 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.745300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.745240 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.745300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.745276 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.745480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.745456 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.747063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.747042 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.747319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.747296 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.770269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.770249 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qpj\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.770408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.770388 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:08.817890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:08.817870 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:09.148311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:09.148277 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9"] Oct 30 15:01:09.149255 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:01:09.149230 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod722a97e4_2863_49b5_a75c_f51b189279f5.slice/crio-3c7da6db6c57ff00a10f781f1891731c554b2995e2abdb9c6241e532da8aafc4 WatchSource:0}: Error finding container 3c7da6db6c57ff00a10f781f1891731c554b2995e2abdb9c6241e532da8aafc4: Status 404 returned error can't find the container with id 3c7da6db6c57ff00a10f781f1891731c554b2995e2abdb9c6241e532da8aafc4 Oct 30 15:01:09.151616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:09.151585 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:01:09.151709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:09.151649 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:01:09.151709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:09.151678 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:01:09.207651 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:01:09.207618 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:01:10.004216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:10.004180 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" event={"ID":"722a97e4-2863-49b5-a75c-f51b189279f5","Type":"ContainerStarted","Data":"ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08"} Oct 30 15:01:10.004636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:10.004260 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" event={"ID":"722a97e4-2863-49b5-a75c-f51b189279f5","Type":"ContainerStarted","Data":"3c7da6db6c57ff00a10f781f1891731c554b2995e2abdb9c6241e532da8aafc4"} Oct 30 15:01:10.084917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:10.084863 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podStartSLOduration=2.084849041 podStartE2EDuration="2.084849041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:10.07475742 +0000 UTC m=+774.033798825" watchObservedRunningTime="2026-10-30 15:01:10.084849041 +0000 UTC m=+774.043890442" Oct 30 15:01:10.822974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:10.822942 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:10.823330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:10.823304 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 15:01:10.823548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:10.823521 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 15:01:11.818627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:11.818594 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 15:01:11.819015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:11.818652 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 15:01:12.818604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:12.818570 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 15:01:12.818778 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:12.818630 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 15:01:13.818668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:13.818637 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 15:01:13.818862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:13.818696 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 15:01:14.818274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:14.818242 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 15:01:14.818457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:14.818301 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 15:01:15.953949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:15.953912 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:15.954710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:15.954690 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:15.955379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:15.955337 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:01:22.417236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.417195 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h"] Oct 30 15:01:22.422814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.422793 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.441478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.441440 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h"] Oct 30 15:01:22.452305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.452280 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:22.452534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.452517 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:22.520892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.520857 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.521022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.520972 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvfhs\" (UniqueName: \"kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.521089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.521037 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.622477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.622451 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvfhs\" (UniqueName: \"kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.623020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.622997 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.623981 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.623958 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.624456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.623869 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.631171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.631150 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.647711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.647693 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvfhs\" (UniqueName: \"kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs\") pod \"maas-api-revoke-sub-c06022e8-27k4h\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:22.743930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:22.743866 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:23.049890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:23.049867 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h"] Oct 30 15:01:23.051305 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:01:23.051279 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad12281b_dbb6_451d_8190_d6bd576cbf25.slice/crio-ed5e802afb69db9f5b3859d75e69842285797bb03b3e69934fc1b0b20db56dda WatchSource:0}: Error finding container ed5e802afb69db9f5b3859d75e69842285797bb03b3e69934fc1b0b20db56dda: Status 404 returned error can't find the container with id ed5e802afb69db9f5b3859d75e69842285797bb03b3e69934fc1b0b20db56dda Oct 30 15:01:23.084459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:23.084426 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" event={"ID":"ad12281b-dbb6-451d-8190-d6bd576cbf25","Type":"ContainerStarted","Data":"ed5e802afb69db9f5b3859d75e69842285797bb03b3e69934fc1b0b20db56dda"} Oct 30 15:01:24.090945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:24.090911 6554 generic.go:356] "Generic (PLEG): container finished" podID="ad12281b-dbb6-451d-8190-d6bd576cbf25" containerID="48ffcab7186d0ea46ef99a7cc7cf6dfd09fb1fc58e6aef286241d48be4720663" exitCode=0 Oct 30 15:01:24.091297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:24.090995 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" event={"ID":"ad12281b-dbb6-451d-8190-d6bd576cbf25","Type":"ContainerDied","Data":"48ffcab7186d0ea46ef99a7cc7cf6dfd09fb1fc58e6aef286241d48be4720663"} Oct 30 15:01:25.267287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.267259 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:25.357031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.356959 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca\") pod \"ad12281b-dbb6-451d-8190-d6bd576cbf25\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " Oct 30 15:01:25.357179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.357093 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca\") pod \"ad12281b-dbb6-451d-8190-d6bd576cbf25\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " Oct 30 15:01:25.357179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.357145 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs\" (UniqueName: \"kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs\") pod \"ad12281b-dbb6-451d-8190-d6bd576cbf25\" (UID: \"ad12281b-dbb6-451d-8190-d6bd576cbf25\") " Oct 30 15:01:25.357533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.357500 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca" pod "ad12281b-dbb6-451d-8190-d6bd576cbf25" (UID: "ad12281b-dbb6-451d-8190-d6bd576cbf25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:25.357685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.357631 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:25.359308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.359270 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca" pod "ad12281b-dbb6-451d-8190-d6bd576cbf25" (UID: "ad12281b-dbb6-451d-8190-d6bd576cbf25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:25.359308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.359290 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs" pod "ad12281b-dbb6-451d-8190-d6bd576cbf25" (UID: "ad12281b-dbb6-451d-8190-d6bd576cbf25"). InnerVolumeSpecName "kube-api-access-zvfhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:25.459179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.459141 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvfhs\" (UniqueName: \"kubernetes.io/projected/ad12281b-dbb6-451d-8190-d6bd576cbf25-kube-api-access-zvfhs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:25.459179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:25.459180 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad12281b-dbb6-451d-8190-d6bd576cbf25-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:26.108262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:26.108220 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" event={"ID":"ad12281b-dbb6-451d-8190-d6bd576cbf25","Type":"ContainerDied","Data":"ed5e802afb69db9f5b3859d75e69842285797bb03b3e69934fc1b0b20db56dda"} Oct 30 15:01:26.108671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:26.108652 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5e802afb69db9f5b3859d75e69842285797bb03b3e69934fc1b0b20db56dda" Oct 30 15:01:26.108815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:26.108593 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h" Oct 30 15:01:30.100357 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.100326 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm"] Oct 30 15:01:30.100894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.100879 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ad12281b-dbb6-451d-8190-d6bd576cbf25" containerName="revoke-keys" Oct 30 15:01:30.100955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.100896 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad12281b-dbb6-451d-8190-d6bd576cbf25" containerName="revoke-keys" Oct 30 15:01:30.101031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.101021 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad12281b-dbb6-451d-8190-d6bd576cbf25" containerName="revoke-keys" Oct 30 15:01:30.104613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.104596 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.125271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.125251 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm"] Oct 30 15:01:30.136497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.136478 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:30.136701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.136680 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:30.235769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.235744 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.235874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.235782 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.235925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.235908 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqkd\" (UniqueName: \"kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.336469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.336444 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.336563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.336488 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.336618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.336592 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqkd\" (UniqueName: \"kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.337169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.337150 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.338796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.338777 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.366952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.366895 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqkd\" (UniqueName: \"kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd\") pod \"maas-api-revoke-sub-f81e8d13-8ldsm\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.416672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.416645 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:30.736808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:30.736768 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm"] Oct 30 15:01:30.746810 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:01:30.746771 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod394db93e_3e6f_42a5_b356_a09ca41483fe.slice/crio-6dd8226d5bc2ba7f6c98b7d29b1f1f33613d699352006f52b0cbe7deb62d3aba WatchSource:0}: Error finding container 6dd8226d5bc2ba7f6c98b7d29b1f1f33613d699352006f52b0cbe7deb62d3aba: Status 404 returned error can't find the container with id 6dd8226d5bc2ba7f6c98b7d29b1f1f33613d699352006f52b0cbe7deb62d3aba Oct 30 15:01:31.138712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:31.138681 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" event={"ID":"394db93e-3e6f-42a5-b356-a09ca41483fe","Type":"ContainerStarted","Data":"42f0f7735d99ec9a8e3f75fc3b7f0fb266b8442479f2adc7b9be9c58092951b9"} Oct 30 15:01:31.139056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:31.138730 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" event={"ID":"394db93e-3e6f-42a5-b356-a09ca41483fe","Type":"ContainerStarted","Data":"6dd8226d5bc2ba7f6c98b7d29b1f1f33613d699352006f52b0cbe7deb62d3aba"} Oct 30 15:01:31.182893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:31.182845 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" podStartSLOduration=1.182831788 podStartE2EDuration="1.182831788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:31.181431586 +0000 UTC m=+795.140472987" watchObservedRunningTime="2026-10-30 15:01:31.182831788 +0000 UTC m=+795.141873192" Oct 30 15:01:32.145916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:32.145881 6554 generic.go:356] "Generic (PLEG): container finished" podID="394db93e-3e6f-42a5-b356-a09ca41483fe" containerID="42f0f7735d99ec9a8e3f75fc3b7f0fb266b8442479f2adc7b9be9c58092951b9" exitCode=0 Oct 30 15:01:32.146343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:32.145961 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" event={"ID":"394db93e-3e6f-42a5-b356-a09ca41483fe","Type":"ContainerDied","Data":"42f0f7735d99ec9a8e3f75fc3b7f0fb266b8442479f2adc7b9be9c58092951b9"} Oct 30 15:01:33.284481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.284458 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:33.368105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.368075 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd\" (UniqueName: \"kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd\") pod \"394db93e-3e6f-42a5-b356-a09ca41483fe\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " Oct 30 15:01:33.368234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.368139 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca\") pod \"394db93e-3e6f-42a5-b356-a09ca41483fe\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " Oct 30 15:01:33.368234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.368178 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca\") pod \"394db93e-3e6f-42a5-b356-a09ca41483fe\" (UID: \"394db93e-3e6f-42a5-b356-a09ca41483fe\") " Oct 30 15:01:33.368602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.368578 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca" pod "394db93e-3e6f-42a5-b356-a09ca41483fe" (UID: "394db93e-3e6f-42a5-b356-a09ca41483fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:33.370132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.370107 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca" pod "394db93e-3e6f-42a5-b356-a09ca41483fe" (UID: "394db93e-3e6f-42a5-b356-a09ca41483fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:33.370223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.370167 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd" pod "394db93e-3e6f-42a5-b356-a09ca41483fe" (UID: "394db93e-3e6f-42a5-b356-a09ca41483fe"). InnerVolumeSpecName "kube-api-access-btqkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:33.469969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.469912 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:33.469969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.469937 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/394db93e-3e6f-42a5-b356-a09ca41483fe-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:33.469969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:33.469950 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqkd\" (UniqueName: \"kubernetes.io/projected/394db93e-3e6f-42a5-b356-a09ca41483fe-kube-api-access-btqkd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:34.157732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:34.157697 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" event={"ID":"394db93e-3e6f-42a5-b356-a09ca41483fe","Type":"ContainerDied","Data":"6dd8226d5bc2ba7f6c98b7d29b1f1f33613d699352006f52b0cbe7deb62d3aba"} Oct 30 15:01:34.157732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:34.157726 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm" Oct 30 15:01:34.157732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:34.157739 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd8226d5bc2ba7f6c98b7d29b1f1f33613d699352006f52b0cbe7deb62d3aba" Oct 30 15:01:39.201701 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:01:39.201664 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:01:39.592428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.592343 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d"] Oct 30 15:01:39.593058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.593037 6554 cpu_manager.go:414] "containerMap: removing container" podUID="394db93e-3e6f-42a5-b356-a09ca41483fe" containerName="revoke-keys" Oct 30 15:01:39.593183 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.593059 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="394db93e-3e6f-42a5-b356-a09ca41483fe" containerName="revoke-keys" Oct 30 15:01:39.593242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.593184 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="394db93e-3e6f-42a5-b356-a09ca41483fe" containerName="revoke-keys" Oct 30 15:01:39.596765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.596741 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.616267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.616249 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-83ff4e-openshift-default-dockercfg-l8gp9\"" Oct 30 15:01:39.632813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.632789 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d"] Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744053 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744121 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744148 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744202 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744251 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744321 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgjqd\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744370 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744399 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.744835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.744492 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846127 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846172 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846191 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846239 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846282 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846342 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgjqd\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846403 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846435 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846530 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846657 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846672 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846814 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.846971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.846943 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.847205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.847164 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.848850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.848830 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.848850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.848836 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.871037 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.871011 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.871645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.871620 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgjqd\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:39.911527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:39.911503 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:40.287377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:40.284284 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d"] Oct 30 15:01:40.341920 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:01:40.337508 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9143c9e0_e90a_4d0f_8874_2b9a3df5f0f6.slice/crio-8acd3a4ca32f00f6ea8d925cb5b11de45b41eda855280d744360ef4ebb6285a4 WatchSource:0}: Error finding container 8acd3a4ca32f00f6ea8d925cb5b11de45b41eda855280d744360ef4ebb6285a4: Status 404 returned error can't find the container with id 8acd3a4ca32f00f6ea8d925cb5b11de45b41eda855280d744360ef4ebb6285a4 Oct 30 15:01:40.341920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:40.341836 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:01:40.342323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:40.341949 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:01:40.342323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:40.341991 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:01:41.225276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:41.225230 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" event={"ID":"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6","Type":"ContainerStarted","Data":"dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd"} Oct 30 15:01:41.225493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:41.225317 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" event={"ID":"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6","Type":"ContainerStarted","Data":"8acd3a4ca32f00f6ea8d925cb5b11de45b41eda855280d744360ef4ebb6285a4"} Oct 30 15:01:41.294685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:41.294620 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podStartSLOduration=2.294599617 podStartE2EDuration="2.294599617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:41.292501441 +0000 UTC m=+805.251542857" watchObservedRunningTime="2026-10-30 15:01:41.294599617 +0000 UTC m=+805.253641024" Oct 30 15:01:41.911680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:41.911651 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:41.913454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:41.913400 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:41.913651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:41.913463 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 15:01:42.912689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:42.912659 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:42.913091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:42.912705 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 15:01:43.911854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:43.911810 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:43.912041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:43.911913 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 15:01:44.912724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:44.912684 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:44.913118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:44.912741 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 15:01:45.912463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:45.912431 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:45.912633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:45.912492 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 15:01:46.911761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:46.911727 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 15:01:46.912275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:46.911789 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 15:01:47.357153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.357067 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg"] Oct 30 15:01:47.368036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.368000 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.382271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.382242 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg"] Oct 30 15:01:47.392944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.392904 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:01:47.393291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.393273 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:01:47.509083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.509055 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.509281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.509104 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.509281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.509198 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psvvw\" (UniqueName: \"kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.610324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.610246 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.610324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.610297 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.610526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.610337 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psvvw\" (UniqueName: \"kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.611038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.611012 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.612553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.612533 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.635337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.635307 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psvvw\" (UniqueName: \"kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw\") pod \"maas-api-revoke-sub-aee54494-nbmxg\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.690059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.690035 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:47.916693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.916670 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:47.917095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.917072 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:47.917750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.917729 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:01:47.991363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:47.991325 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg"] Oct 30 15:01:47.994498 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:01:47.994473 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a2fbdf_9d64_421a_afb3_38bc25bd0e9c.slice/crio-e5dee064f1fdff60fae13fc04ac481236f59ec03c760ada33d709fcc7a4650da WatchSource:0}: Error finding container e5dee064f1fdff60fae13fc04ac481236f59ec03c760ada33d709fcc7a4650da: Status 404 returned error can't find the container with id e5dee064f1fdff60fae13fc04ac481236f59ec03c760ada33d709fcc7a4650da Oct 30 15:01:48.304575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.299538 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" event={"ID":"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c","Type":"ContainerStarted","Data":"95f71394717eb41a7bc676f20b1542761b63c02422f17c61cc119bd85644db4d"} Oct 30 15:01:48.304575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.299594 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" event={"ID":"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c","Type":"ContainerStarted","Data":"e5dee064f1fdff60fae13fc04ac481236f59ec03c760ada33d709fcc7a4650da"} Oct 30 15:01:48.345550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.345510 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" podStartSLOduration=1.34549664 podStartE2EDuration="1.34549664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:48.343862685 +0000 UTC m=+812.302904093" watchObservedRunningTime="2026-10-30 15:01:48.34549664 +0000 UTC m=+812.304538045" Oct 30 15:01:48.371471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.371444 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf"] Oct 30 15:01:48.376133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.376114 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.393913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.393868 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"openshift-service-ca.crt\"" Oct 30 15:01:48.394647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.394623 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"default-dockercfg-576h8\"" Oct 30 15:01:48.394750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.394692 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"kube-root-ca.crt\"" Oct 30 15:01:48.394750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.394710 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-357948\"/\"distinct-w3-357948-kserve-self-signed-certs\"" Oct 30 15:01:48.395862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.395839 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf"] Oct 30 15:01:48.519237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.519178 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.519237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.519220 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.519399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.519239 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.519399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.519340 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t75t\" (UniqueName: \"kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.519495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.519472 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.519604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.519590 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.620671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.620625 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.620849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.620680 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.620849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.620706 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.620849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.620752 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t75t\" (UniqueName: \"kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.620849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.620842 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.621061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.620956 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.621122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.621059 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.621180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.621119 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.621365 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.621261 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.623089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.623068 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.623581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.623565 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.645769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.645738 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t75t\" (UniqueName: \"kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t\") pod \"distinct-w3-357948-kserve-86c4db5f86-2lshf\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.691613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.691591 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:48.993998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:48.993960 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf"] Oct 30 15:01:49.307316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:49.307275 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" event={"ID":"cb90950f-4c23-4af9-b33a-4408a5d1ec68","Type":"ContainerStarted","Data":"be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9"} Oct 30 15:01:49.307521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:49.307327 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" event={"ID":"cb90950f-4c23-4af9-b33a-4408a5d1ec68","Type":"ContainerStarted","Data":"aea9ee83e9ca64833c9016fa55ddf79c615c13201b4364b4819bdaac090a3d2e"} Oct 30 15:01:49.309230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:49.309200 6554 generic.go:356] "Generic (PLEG): container finished" podID="10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" containerID="95f71394717eb41a7bc676f20b1542761b63c02422f17c61cc119bd85644db4d" exitCode=0 Oct 30 15:01:49.309402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:49.309285 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" event={"ID":"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c","Type":"ContainerDied","Data":"95f71394717eb41a7bc676f20b1542761b63c02422f17c61cc119bd85644db4d"} Oct 30 15:01:49.357762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:49.357718 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podStartSLOduration=1.3577055279999999 podStartE2EDuration="1.357705528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:01:49.355731416 +0000 UTC m=+813.314772834" watchObservedRunningTime="2026-10-30 15:01:49.357705528 +0000 UTC m=+813.316746927" Oct 30 15:01:49.691945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:49.691914 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:50.453042 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.453019 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:50.542026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.541998 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca\") pod \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " Oct 30 15:01:50.542144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.542114 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca\") pod \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " Oct 30 15:01:50.542244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.542229 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw\" (UniqueName: \"kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw\") pod \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\" (UID: \"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c\") " Oct 30 15:01:50.542488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.542462 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca" pod "10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" (UID: "10a2fbdf-9d64-421a-afb3-38bc25bd0e9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:01:50.542706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.542687 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:50.544126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.544097 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca" pod "10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" (UID: "10a2fbdf-9d64-421a-afb3-38bc25bd0e9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:01:50.544714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.544691 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw" pod "10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" (UID: "10a2fbdf-9d64-421a-afb3-38bc25bd0e9c"). InnerVolumeSpecName "kube-api-access-psvvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:01:50.643917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.643896 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psvvw\" (UniqueName: \"kubernetes.io/projected/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-kube-api-access-psvvw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:50.643917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:50.643917 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:01:51.322077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:51.322044 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" Oct 30 15:01:51.322257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:51.322075 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg" event={"ID":"10a2fbdf-9d64-421a-afb3-38bc25bd0e9c","Type":"ContainerDied","Data":"e5dee064f1fdff60fae13fc04ac481236f59ec03c760ada33d709fcc7a4650da"} Oct 30 15:01:51.322257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:51.322121 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5dee064f1fdff60fae13fc04ac481236f59ec03c760ada33d709fcc7a4650da" Oct 30 15:01:58.692542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:58.692454 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:58.705250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:58.705219 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:01:59.385993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:01:59.385958 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:02:07.530737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.530701 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl"] Oct 30 15:02:07.531720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.531697 6554 cpu_manager.go:414] "containerMap: removing container" podUID="10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" containerName="revoke-keys" Oct 30 15:02:07.531793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.531725 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" containerName="revoke-keys" Oct 30 15:02:07.531935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.531921 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" containerName="revoke-keys" Oct 30 15:02:07.536316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.536297 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.568718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.568696 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"distinct-w6-2c8d89-kserve-self-signed-certs\"" Oct 30 15:02:07.570913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.570888 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl"] Oct 30 15:02:07.617123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.617097 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvfd\" (UniqueName: \"kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.617221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.617148 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.617260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.617219 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.617385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.617370 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.617437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.617413 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.617471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.617450 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.720665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.720627 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.720665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.720677 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.720996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.720726 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.720996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.720773 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvfd\" (UniqueName: \"kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.720996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.720835 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.720996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.720865 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.721454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.721391 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.721942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.721881 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.722476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.722432 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.750516 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.750476 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.759934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.759902 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.760395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.760337 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvfd\" (UniqueName: \"kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd\") pod \"distinct-w6-2c8d89-kserve-69659f7f75-wdkxl\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:07.849569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:07.849497 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:08.154995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.154968 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl"] Oct 30 15:02:08.156000 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:08.155972 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89ca9f3c_fcaf_4848_b91e_a04f8e9352f0.slice/crio-436a7729601f6f6ae0d6bd03c1bb5b9e6401ca060447ae68fa5e7f2d29211169 WatchSource:0}: Error finding container 436a7729601f6f6ae0d6bd03c1bb5b9e6401ca060447ae68fa5e7f2d29211169: Status 404 returned error can't find the container with id 436a7729601f6f6ae0d6bd03c1bb5b9e6401ca060447ae68fa5e7f2d29211169 Oct 30 15:02:08.428920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.428840 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" event={"ID":"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0","Type":"ContainerStarted","Data":"91b55505b5c04a2303a8f8e00fde6e2e7cd65ac5b8b3eec65f8b17c24af315c4"} Oct 30 15:02:08.428920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.428883 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" event={"ID":"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0","Type":"ContainerStarted","Data":"436a7729601f6f6ae0d6bd03c1bb5b9e6401ca060447ae68fa5e7f2d29211169"} Oct 30 15:02:08.475339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.475287 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podStartSLOduration=1.475274473 podStartE2EDuration="1.475274473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:08.473432981 +0000 UTC m=+832.432474388" watchObservedRunningTime="2026-10-30 15:02:08.475274473 +0000 UTC m=+832.434315892" Oct 30 15:02:08.844422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.844307 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw"] Oct 30 15:02:08.848608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.848584 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.849733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.849712 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:08.884019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.883990 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw"] Oct 30 15:02:08.937421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937391 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937469 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937534 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937575 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937662 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937703 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvnv\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937731 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937806 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:08.937878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:08.937860 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039362 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039413 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039464 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039538 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039566 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039602 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039643 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039698 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvnv\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039730 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.039969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.039952 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.040169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.040142 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.040267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.040198 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.040398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.040342 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.040525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.040502 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.042424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.042404 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.049905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.049397 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.065773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.065746 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvnv\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.065940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.065923 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.163309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.163286 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:09.203218 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:02:09.203187 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:02:09.242850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.242798 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl"] Oct 30 15:02:09.248852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.248823 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.269129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.269103 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl"] Oct 30 15:02:09.342918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.342891 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.342940 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343003 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343053 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343088 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9nv2\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343109 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343157 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343238 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.343317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.343300 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.444832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.444774 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.444832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.444830 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.444985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.444865 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.444985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.444889 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.444985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.444926 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.444985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.444980 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9nv2\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445015 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445043 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445076 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445196 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445255 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445282 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445431 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.445763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.445742 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.447194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.447170 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.453988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.447729 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.469669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.469649 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9nv2\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.470469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.470451 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.483011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.482993 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw"] Oct 30 15:02:09.484384 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:09.484363 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce60949b_1edc_45a1_b35a_10b694575297.slice/crio-6508ab37c204827080d5c5f532e569473a757fdf84ee6f740dc9906605353669 WatchSource:0}: Error finding container 6508ab37c204827080d5c5f532e569473a757fdf84ee6f740dc9906605353669: Status 404 returned error can't find the container with id 6508ab37c204827080d5c5f532e569473a757fdf84ee6f740dc9906605353669 Oct 30 15:02:09.486711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.486682 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:09.486789 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.486739 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:09.486789 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.486767 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:09.564665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.564643 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:09.888961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.888937 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl"] Oct 30 15:02:09.890063 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:09.890033 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7174b8_c9bc_4d5b_939f_f1a1ebf4511c.slice/crio-ddb8038bc5be1c6a7421f947478f75d8a3121b005bdbc49c8244f8651bac9a84 WatchSource:0}: Error finding container ddb8038bc5be1c6a7421f947478f75d8a3121b005bdbc49c8244f8651bac9a84: Status 404 returned error can't find the container with id ddb8038bc5be1c6a7421f947478f75d8a3121b005bdbc49c8244f8651bac9a84 Oct 30 15:02:09.892666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.892636 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:09.892735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.892719 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:09.892785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:09.892749 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:10.443623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.443575 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" event={"ID":"ce60949b-1edc-45a1-b35a-10b694575297","Type":"ContainerStarted","Data":"d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b"} Oct 30 15:02:10.443831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.443657 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" event={"ID":"ce60949b-1edc-45a1-b35a-10b694575297","Type":"ContainerStarted","Data":"6508ab37c204827080d5c5f532e569473a757fdf84ee6f740dc9906605353669"} Oct 30 15:02:10.445273 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.445247 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" event={"ID":"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c","Type":"ContainerStarted","Data":"366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2"} Oct 30 15:02:10.445412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.445296 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" event={"ID":"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c","Type":"ContainerStarted","Data":"ddb8038bc5be1c6a7421f947478f75d8a3121b005bdbc49c8244f8651bac9a84"} Oct 30 15:02:10.527050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.526996 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podStartSLOduration=2.526982408 podStartE2EDuration="2.526982408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:10.511138949 +0000 UTC m=+834.470180354" watchObservedRunningTime="2026-10-30 15:02:10.526982408 +0000 UTC m=+834.486023812" Oct 30 15:02:10.577699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.571434 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:10.583494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.581460 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:10.583494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:10.581517 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:11.171372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:11.169677 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:11.187483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:11.183764 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:11.187483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:11.183838 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:02:11.579431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:11.577483 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:11.579431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:11.577563 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:12.177375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:12.175453 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:12.177375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:12.175511 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:02:12.566430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:12.565682 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:12.566430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:12.565741 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:13.174374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:13.173583 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:13.174374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:13.173653 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:02:13.573432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:13.572713 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:13.573432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:13.572769 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:14.181380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:14.180491 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:14.181380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:14.180568 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:02:14.576435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:14.575717 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:14.576435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:14.575786 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:15.168374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:15.167711 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:15.168374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:15.167771 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:02:15.569433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:15.568728 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:15.569433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:15.568787 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:16.165374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:16.164660 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:02:16.165374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:16.164716 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:02:16.567109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:16.566575 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 15:02:16.567109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:16.566637 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 15:02:17.319373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:17.318982 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:17.331371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:17.321808 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:17.331371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:17.330099 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:02:17.471374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:17.470400 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podStartSLOduration=8.449333983 podStartE2EDuration="8.449333983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:10.572143668 +0000 UTC m=+834.531185081" watchObservedRunningTime="2026-10-30 15:02:17.449333983 +0000 UTC m=+841.408375389" Oct 30 15:02:17.884141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:17.884108 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:18.077279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.077247 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:18.640737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.640699 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:02:18.641190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.640763 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:02:18.663982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.663957 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:02:18.917306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.917215 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:18.917644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.917613 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:18.919258 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:18.919228 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:02:19.364526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.364447 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg"] Oct 30 15:02:19.369838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.369817 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.386667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.386650 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"distinct-w1-83ff4e-kserve-self-signed-certs\"" Oct 30 15:02:19.389612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.389593 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"openshift-service-ca.crt\"" Oct 30 15:02:19.389711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.389651 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"kube-root-ca.crt\"" Oct 30 15:02:19.389711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.389690 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg"] Oct 30 15:02:19.464997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.464975 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.465102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.465002 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.465102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.465038 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwtl\" (UniqueName: \"kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.465222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.465187 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.465389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.465343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.465458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.465435 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.486932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.486916 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-83ff4e\"/\"default-dockercfg-6nljl\"" Oct 30 15:02:19.567032 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.566989 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.567224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.567043 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.567224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.567099 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.567224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.567120 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.567224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.567171 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwtl\" (UniqueName: \"kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.567224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.567222 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.568120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.567831 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.568120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.568080 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.584380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.571157 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.584380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.572326 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.584380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.575836 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.593575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.593537 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwtl\" (UniqueName: \"kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl\") pod \"distinct-w1-83ff4e-kserve-5cdb458d94-7srzg\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:19.682836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:19.682808 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:20.009302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:20.009269 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg"] Oct 30 15:02:20.009982 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:20.009957 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc01767e6_485f_4aa7_bcde_9ab19983e054.slice/crio-31aab81ad715f8884517412eb6fe8621a381ed2621309553d4b2ad98d66585eb WatchSource:0}: Error finding container 31aab81ad715f8884517412eb6fe8621a381ed2621309553d4b2ad98d66585eb: Status 404 returned error can't find the container with id 31aab81ad715f8884517412eb6fe8621a381ed2621309553d4b2ad98d66585eb Oct 30 15:02:20.665566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:20.665534 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" event={"ID":"c01767e6-485f-4aa7-bcde-9ab19983e054","Type":"ContainerStarted","Data":"1b2ed7a3e3c577a9570d6db784a8c8faab63e2313e59cf7ccd2365e1e5aeaf41"} Oct 30 15:02:20.665734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:20.665574 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" event={"ID":"c01767e6-485f-4aa7-bcde-9ab19983e054","Type":"ContainerStarted","Data":"31aab81ad715f8884517412eb6fe8621a381ed2621309553d4b2ad98d66585eb"} Oct 30 15:02:20.683123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:20.683098 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:20.711369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:20.711308 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podStartSLOduration=1.7112969900000001 podStartE2EDuration="1.71129699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:20.710253665 +0000 UTC m=+844.669295071" watchObservedRunningTime="2026-10-30 15:02:20.71129699 +0000 UTC m=+844.670338396" Oct 30 15:02:29.683990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.683952 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:29.696195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.696174 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:29.729011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.728992 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:02:29.956738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.956655 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25"] Oct 30 15:02:29.973157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.973123 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:29.980284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.980253 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25"] Oct 30 15:02:29.989786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:29.989765 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-4f5c30\"/\"distinct-w5-4f5c30-kserve-self-signed-certs\"" Oct 30 15:02:30.074842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.074812 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.074951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.074877 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.074990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.074950 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.075077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.075062 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khk9s\" (UniqueName: \"kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.075114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.075099 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.075152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.075127 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.176496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.176452 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.176852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.176825 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.176990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.176976 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.177262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.177241 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khk9s\" (UniqueName: \"kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.177452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.177425 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.177614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.177565 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.177800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.177777 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.178072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.178047 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.178715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.177263 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.180242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.180218 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.181412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.181392 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.203213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.203192 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khk9s\" (UniqueName: \"kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s\") pod \"distinct-w5-4f5c30-kserve-55b7b79749-tvb25\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.289577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.289522 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:30.592019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.591988 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25"] Oct 30 15:02:30.592568 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:30.592538 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05fd71f9_b19f_4a9d_8c80_e2097316161d.slice/crio-e4173517216d5d399c9364bc01e168e137729c1a3b0124c98d4f5282457dbd7d WatchSource:0}: Error finding container e4173517216d5d399c9364bc01e168e137729c1a3b0124c98d4f5282457dbd7d: Status 404 returned error can't find the container with id e4173517216d5d399c9364bc01e168e137729c1a3b0124c98d4f5282457dbd7d Oct 30 15:02:30.727069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.727036 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" event={"ID":"05fd71f9-b19f-4a9d-8c80-e2097316161d","Type":"ContainerStarted","Data":"61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc"} Oct 30 15:02:30.727069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.727079 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" event={"ID":"05fd71f9-b19f-4a9d-8c80-e2097316161d","Type":"ContainerStarted","Data":"e4173517216d5d399c9364bc01e168e137729c1a3b0124c98d4f5282457dbd7d"} Oct 30 15:02:30.774076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:30.774032 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podStartSLOduration=1.7740178960000001 podStartE2EDuration="1.774017896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:30.772783487 +0000 UTC m=+854.731824895" watchObservedRunningTime="2026-10-30 15:02:30.774017896 +0000 UTC m=+854.733059300" Oct 30 15:02:31.290283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:31.290250 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:39.201622 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:02:39.201581 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:02:39.480334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.480257 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc"] Oct 30 15:02:39.484226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.484210 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.518767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.518736 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc"] Oct 30 15:02:39.521746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.521722 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.521870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.521758 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.521870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.521788 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.522076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.521896 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.522076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.521925 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzbdp\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.522076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.522007 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.522076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.522050 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.522290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.522119 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.522290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.522176 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623264 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623400 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623437 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623467 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623513 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623563 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzbdp\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623646 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623687 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.623885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623760 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.624007 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.623889 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.624076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.624034 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.624076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.624039 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.624196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.624104 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.624196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.624145 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.625724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.625702 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.626029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.626006 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.654771 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.654746 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.662208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.662186 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzbdp\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp\") pod \"e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:39.800702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:39.800647 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:40.140984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.140955 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc"] Oct 30 15:02:40.141608 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:40.141582 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33c93d8_f005_4910_b132_f02662ec3596.slice/crio-4e1a815979e343445c08d7f1f8718ba7c5711a2f8804775576ca046fcb658d87 WatchSource:0}: Error finding container 4e1a815979e343445c08d7f1f8718ba7c5711a2f8804775576ca046fcb658d87: Status 404 returned error can't find the container with id 4e1a815979e343445c08d7f1f8718ba7c5711a2f8804775576ca046fcb658d87 Oct 30 15:02:40.143919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.143885 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:40.143984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.143949 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:40.143984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.143977 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:40.290397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.290362 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:40.302744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.302722 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:40.812567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.812533 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" event={"ID":"a33c93d8-f005-4910-b132-f02662ec3596","Type":"ContainerStarted","Data":"255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485"} Oct 30 15:02:40.812758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.812587 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" event={"ID":"a33c93d8-f005-4910-b132-f02662ec3596","Type":"ContainerStarted","Data":"4e1a815979e343445c08d7f1f8718ba7c5711a2f8804775576ca046fcb658d87"} Oct 30 15:02:40.823684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.823659 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:02:40.880597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:40.880541 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podStartSLOduration=1.880525926 podStartE2EDuration="1.880525926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:40.879430181 +0000 UTC m=+864.838471586" watchObservedRunningTime="2026-10-30 15:02:40.880525926 +0000 UTC m=+864.839567331" Oct 30 15:02:41.801015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:41.800976 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:41.802824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:41.802796 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 15:02:41.802945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:41.802848 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 15:02:42.801769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:42.801737 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 15:02:42.802368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:42.801802 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 15:02:43.801752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:43.801721 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 15:02:43.801925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:43.801773 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 15:02:44.801766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:44.801728 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 15:02:44.802002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:44.801811 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 15:02:45.782371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.781973 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4"] Oct 30 15:02:45.788855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.788816 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.803956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.803819 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 15:02:45.806772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.806748 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 15:02:45.807716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.807692 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4"] Oct 30 15:02:45.809458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.809437 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-2c8d89\"/\"distinct-2-w6-2c8d89-kserve-self-signed-certs\"" Oct 30 15:02:45.822162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.822136 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.822381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.822333 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.822465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.822444 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.822577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.822561 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.822620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.822607 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.822744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.822723 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj8tg\" (UniqueName: \"kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924167 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924290 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924335 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924443 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924493 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924546 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj8tg\" (UniqueName: \"kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924807 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924872 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.924968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.924905 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.926721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.926700 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.926977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.926961 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:45.950049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:45.950031 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj8tg\" (UniqueName: \"kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg\") pod \"distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:46.112853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.112762 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:46.423082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.423058 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4"] Oct 30 15:02:46.424550 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:46.424524 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf350623a_6add_4f8d_a000_de0cc57ca3af.slice/crio-fb39395f74a49508d31629c61892be4cf1a4bb0c3a082894da4a71eb18fe3549 WatchSource:0}: Error finding container fb39395f74a49508d31629c61892be4cf1a4bb0c3a082894da4a71eb18fe3549: Status 404 returned error can't find the container with id fb39395f74a49508d31629c61892be4cf1a4bb0c3a082894da4a71eb18fe3549 Oct 30 15:02:46.848835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.848687 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:46.849402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.849232 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:46.850382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.850325 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:02:46.852379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.852334 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" event={"ID":"f350623a-6add-4f8d-a000-de0cc57ca3af","Type":"ContainerStarted","Data":"3c7c8d4f12ac86127b1bd4085094b93aa32bbfc70abcb3057660b173668e5087"} Oct 30 15:02:46.852490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.852389 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" event={"ID":"f350623a-6add-4f8d-a000-de0cc57ca3af","Type":"ContainerStarted","Data":"fb39395f74a49508d31629c61892be4cf1a4bb0c3a082894da4a71eb18fe3549"} Oct 30 15:02:46.960155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:46.960114 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podStartSLOduration=1.960103169 podStartE2EDuration="1.960103169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:46.958460088 +0000 UTC m=+870.917501493" watchObservedRunningTime="2026-10-30 15:02:46.960103169 +0000 UTC m=+870.919144598" Oct 30 15:02:47.112921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:47.112854 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:53.991323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:53.991291 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms"] Oct 30 15:02:53.996237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:53.996215 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.021661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.021636 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms"] Oct 30 15:02:54.125084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125045 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125182 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125289 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125373 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pltrm\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125511 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125547 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125653 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125755 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.125891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.125862 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.226768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.226723 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.226768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.226768 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.226848 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.226876 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.226924 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.226986 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.227031 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.227056 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.227111 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pltrm\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.227751 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.227951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.227934 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.228371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.228020 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.228490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.228258 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.228577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.228336 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.231587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.231556 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.232940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.232915 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.252831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.252753 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pltrm\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.254172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.254138 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token\") pod \"e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.311600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.311568 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:54.670224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.669736 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms"] Oct 30 15:02:54.672389 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:54.672334 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c397e63_4528_4cbd_8ad0_30b693ca98f8.slice/crio-560df032776b2325dbd1577266233a309bf8009c07f2a2ed2884b0b8b212dd35 WatchSource:0}: Error finding container 560df032776b2325dbd1577266233a309bf8009c07f2a2ed2884b0b8b212dd35: Status 404 returned error can't find the container with id 560df032776b2325dbd1577266233a309bf8009c07f2a2ed2884b0b8b212dd35 Oct 30 15:02:54.676427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.676385 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:54.676536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.676473 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:54.676536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.676521 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:54.862373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.853546 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2"] Oct 30 15:02:54.866374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.865180 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.939426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.931595 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" event={"ID":"9c397e63-4528-4cbd-8ad0-30b693ca98f8","Type":"ContainerStarted","Data":"b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d"} Oct 30 15:02:54.939426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.931662 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" event={"ID":"9c397e63-4528-4cbd-8ad0-30b693ca98f8","Type":"ContainerStarted","Data":"560df032776b2325dbd1577266233a309bf8009c07f2a2ed2884b0b8b212dd35"} Oct 30 15:02:54.939426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.932287 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2"] Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948078 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948126 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948160 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948203 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948361 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zf246\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948398 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948590 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948651 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:54.958402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:54.948722 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.004408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.004320 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podStartSLOduration=2.004300532 podStartE2EDuration="2.004300532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:55.001968408 +0000 UTC m=+878.961009815" watchObservedRunningTime="2026-10-30 15:02:55.004300532 +0000 UTC m=+878.963341940" Oct 30 15:02:55.050100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050049 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050135 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050162 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050192 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050234 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050412 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zf246\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050471 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050571 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.050754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050628 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.051762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.050985 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.051762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.051155 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.051762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.051508 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.051762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.051557 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.051762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.051714 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.054696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.053771 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.056087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.056060 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.080624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.080595 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zf246\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.082513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.082466 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.251472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.251398 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:55.318731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.318697 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:02:55.322835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.321471 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:02:55.322835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.321527 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:02:55.600430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.600232 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2"] Oct 30 15:02:55.605261 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:02:55.605231 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fe275d_7a84_478c_863d_2db9076587f4.slice/crio-1994ac9572432a3ee3f4a85ad41da51dfcdcc809e24e227550134a633951293c WatchSource:0}: Error finding container 1994ac9572432a3ee3f4a85ad41da51dfcdcc809e24e227550134a633951293c: Status 404 returned error can't find the container with id 1994ac9572432a3ee3f4a85ad41da51dfcdcc809e24e227550134a633951293c Oct 30 15:02:55.607881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.607843 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:55.607998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.607918 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:55.607998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.607945 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:02:55.940155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.940116 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" event={"ID":"21fe275d-7a84-478c-863d-2db9076587f4","Type":"ContainerStarted","Data":"8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2"} Oct 30 15:02:55.940385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:55.940180 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" event={"ID":"21fe275d-7a84-478c-863d-2db9076587f4","Type":"ContainerStarted","Data":"1994ac9572432a3ee3f4a85ad41da51dfcdcc809e24e227550134a633951293c"} Oct 30 15:02:56.019755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.019687 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podStartSLOduration=2.019671997 podStartE2EDuration="2.019671997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:02:56.017162024 +0000 UTC m=+879.976203441" watchObservedRunningTime="2026-10-30 15:02:56.019671997 +0000 UTC m=+879.978713401" Oct 30 15:02:56.113217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.113186 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:56.127024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.127000 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:56.252381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.252273 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:02:56.275375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.253421 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:02:56.275375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.253473 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:02:56.312255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.312211 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:02:56.312442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:56.312286 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:02:57.007025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:57.006883 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:02:57.252071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:57.252035 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:02:57.252449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:57.252090 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:02:57.312805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:57.312731 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:02:57.312805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:57.312784 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:02:58.252572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:58.252538 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:02:58.253031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:58.252595 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:02:58.311959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:58.311925 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:02:58.312118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:58.311973 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:02:59.251773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:59.251736 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:02:59.251985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:59.251789 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:02:59.312980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:59.312942 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:02:59.313601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:02:59.313563 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:03:00.252101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:00.252067 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:03:00.252315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:00.252135 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:03:00.312700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:00.312666 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:03:00.312855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:00.312715 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:03:01.252362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:01.252315 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:03:01.252746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:01.252401 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:03:01.312819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:01.312785 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 15:03:01.312971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:01.312833 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 15:03:02.252403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:02.252338 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 15:03:02.252802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:02.252441 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 15:03:02.548505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:02.548411 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:03:02.548999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:02.548958 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:03:02.549700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:02.549672 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:03:03.256699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:03.256673 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:03:03.257087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:03.257070 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:03:03.257847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:03.257826 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:03:08.863126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:08.863092 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl"] Oct 30 15:03:08.867681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:08.867661 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:08.903589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:08.903551 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl"] Oct 30 15:03:09.014123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014094 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014140 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014188 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh649\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014719 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014763 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014853 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014890 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.014991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.014974 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.015037 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.015020 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.115943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.115871 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.115943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.115909 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.115943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.115935 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh649\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116011 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116041 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116080 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116105 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116165 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116208 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116256 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116401 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116470 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116543 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.116868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.116850 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.118445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.118424 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.118637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.118619 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.140664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.140643 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.140773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.140753 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh649\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.178050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.178023 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz"] Oct 30 15:03:09.181117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.181092 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:09.183848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.183830 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.212273 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.212245 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz"] Oct 30 15:03:09.217474 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:03:09.212928 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:03:09.217474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.216994 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t"] Oct 30 15:03:09.222251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.222229 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.252480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.247188 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t"] Oct 30 15:03:09.318336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318298 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.318510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318395 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.318598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318560 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.318649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318599 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.318691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318668 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.318731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318716 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psndj\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.318776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318750 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.318776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318768 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.318971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318945 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.319025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.318995 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.319073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319054 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.319113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319093 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.319159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319114 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.319159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319132 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.319248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319189 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.319318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319297 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhm9j\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.319390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319336 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.319448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.319426 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.420659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420635 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.420773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420666 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.420773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420683 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.420773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420698 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psndj\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.420773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420724 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.420773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420741 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420816 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420860 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420889 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.420941 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421089 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421139 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421208 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421331 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421384 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421416 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421475 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhm9j\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421507 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421560 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421570 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421604 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.421870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421675 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421683 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.421974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.421930 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.422283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.422026 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.422283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.422145 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.422283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.422272 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.422521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.422444 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.423789 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.423767 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.423934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.423914 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.424062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.424043 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.424115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.424093 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.445486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.445462 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.445632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.445608 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psndj\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.445699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.445642 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhm9j\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.445792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.445775 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.500003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.499983 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:09.528834 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:09.528809 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e39c5bd_d7e5_4342_9663_12b1237020f2.slice/crio-ff6e474d835a31c32068d7b6483655d9841bbb564ef66e72d76c8b0b532aa60c WatchSource:0}: Error finding container ff6e474d835a31c32068d7b6483655d9841bbb564ef66e72d76c8b0b532aa60c: Status 404 returned error can't find the container with id ff6e474d835a31c32068d7b6483655d9841bbb564ef66e72d76c8b0b532aa60c Oct 30 15:03:09.529088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.529069 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl"] Oct 30 15:03:09.531192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.531159 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.531276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.531224 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.531276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.531252 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.551936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.551451 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:09.824498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.824475 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz"] Oct 30 15:03:09.824715 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:09.824689 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4767fe86_4975_44bb_a86f_bafa6b10f05a.slice/crio-66c17d72f1b951d22b0317c9dcbf2a8da1a56c1ad1098d23cd4b348f6d1dfd53 WatchSource:0}: Error finding container 66c17d72f1b951d22b0317c9dcbf2a8da1a56c1ad1098d23cd4b348f6d1dfd53: Status 404 returned error can't find the container with id 66c17d72f1b951d22b0317c9dcbf2a8da1a56c1ad1098d23cd4b348f6d1dfd53 Oct 30 15:03:09.828164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.828125 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.828262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.828210 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.828308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.828259 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.880782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.880760 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t"] Oct 30 15:03:09.884746 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:09.884689 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc78931_2fc6_42b5_94a1_8a2d98513cf6.slice/crio-44338e28782d4efe2117eb0b742734b14af4308578635b2a96a02d9843614712 WatchSource:0}: Error finding container 44338e28782d4efe2117eb0b742734b14af4308578635b2a96a02d9843614712: Status 404 returned error can't find the container with id 44338e28782d4efe2117eb0b742734b14af4308578635b2a96a02d9843614712 Oct 30 15:03:09.887302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.887270 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.887420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.887336 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:09.887491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:09.887423 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:10.080695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.080596 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" event={"ID":"8e39c5bd-d7e5-4342-9663-12b1237020f2","Type":"ContainerStarted","Data":"2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3"} Oct 30 15:03:10.080695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.080690 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" event={"ID":"8e39c5bd-d7e5-4342-9663-12b1237020f2","Type":"ContainerStarted","Data":"ff6e474d835a31c32068d7b6483655d9841bbb564ef66e72d76c8b0b532aa60c"} Oct 30 15:03:10.082730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.082704 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" event={"ID":"3fc78931-2fc6-42b5-94a1-8a2d98513cf6","Type":"ContainerStarted","Data":"2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09"} Oct 30 15:03:10.082863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.082747 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" event={"ID":"3fc78931-2fc6-42b5-94a1-8a2d98513cf6","Type":"ContainerStarted","Data":"44338e28782d4efe2117eb0b742734b14af4308578635b2a96a02d9843614712"} Oct 30 15:03:10.084439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.084416 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" event={"ID":"4767fe86-4975-44bb-a86f-bafa6b10f05a","Type":"ContainerStarted","Data":"6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a"} Oct 30 15:03:10.084547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.084451 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" event={"ID":"4767fe86-4975-44bb-a86f-bafa6b10f05a","Type":"ContainerStarted","Data":"66c17d72f1b951d22b0317c9dcbf2a8da1a56c1ad1098d23cd4b348f6d1dfd53"} Oct 30 15:03:10.138792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.138733 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podStartSLOduration=2.138716568 podStartE2EDuration="2.138716568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:10.138683634 +0000 UTC m=+894.097725040" watchObservedRunningTime="2026-10-30 15:03:10.138716568 +0000 UTC m=+894.097757975" Oct 30 15:03:10.181239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.181196 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:10.182980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.182946 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:10.183094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.183008 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:10.191158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.191097 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podStartSLOduration=1.1910774 podStartE2EDuration="1.1910774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:10.187025607 +0000 UTC m=+894.146067014" watchObservedRunningTime="2026-10-30 15:03:10.1910774 +0000 UTC m=+894.150118807" Oct 30 15:03:10.239643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.239592 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podStartSLOduration=1.239575117 podStartE2EDuration="1.239575117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:10.236017942 +0000 UTC m=+894.195059348" watchObservedRunningTime="2026-10-30 15:03:10.239575117 +0000 UTC m=+894.198616522" Oct 30 15:03:10.506373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.502542 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:10.506373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.502698 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:10.506373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.502763 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:10.552323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.552244 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:10.569382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.566451 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:10.569382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:10.566514 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:11.212373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:11.190766 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:11.212373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:11.190839 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:11.519848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:11.519758 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:11.520121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:11.520090 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:11.551916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:11.551875 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:11.552056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:11.551951 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:12.199374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:12.198366 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:12.199374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:12.198428 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:12.507433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:12.506746 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:12.507433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:12.506821 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:12.552209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:12.552058 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:12.552209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:12.552107 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:13.184377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:13.183802 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:13.184377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:13.183863 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:13.504431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:13.503811 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:13.504431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:13.503879 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:13.552246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:13.552212 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:13.552646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:13.552266 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:14.201373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:14.200874 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:14.201373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:14.200948 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:14.501869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:14.501778 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:14.502138 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:14.502108 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:14.552177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:14.552149 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:14.552337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:14.552200 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:15.185762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:15.185724 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:15.186095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:15.186067 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:15.504432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:15.503717 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:15.504432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:15.503780 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:15.552371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:15.552318 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:15.552794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:15.552396 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:16.191376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.183552 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:16.191905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.191867 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:16.504086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.503993 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:16.504395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.504338 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:16.582383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.570461 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:16.582383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.570539 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:16.824427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.823439 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:03:16.885383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.878444 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:16.981374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:16.960754 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:17.182374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.182180 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 15:03:17.182374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.182272 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 15:03:17.513813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.513721 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 15:03:17.514052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.514021 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 15:03:17.572383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.567894 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 15:03:17.572383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.567961 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 15:03:17.658373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.655424 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:17.844429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.837442 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:17.844429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.838034 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:03:17.844429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.841501 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:17.844429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.843768 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:17.882373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.881521 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:17.882373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:17.882191 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:03:18.256736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.256688 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 15:03:18.404382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.400458 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.404382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.401077 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:18.404382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.401119 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.459376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.459049 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:03:18.730469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.730426 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:03:18.747694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.733220 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:03:18.762122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.733919 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:18.762122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.748018 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:03:18.762122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.749591 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:18.773578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.773554 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:03:18.789458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.789427 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:18.789683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.789653 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.870750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.870710 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.871416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.871392 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:03:18.890370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.872713 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:03:18.894114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.842663 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.894607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.890403 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:18.911403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.898429 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:03:18.911403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.890711 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 15:03:18.911403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:18.899154 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:03:19.096637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.086399 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:19.096637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.086470 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.293476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.293437 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.396384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.393595 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.423264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.421989 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.423264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.422617 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:03:19.455135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.455105 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:03:19.643742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.643714 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:03:19.700926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.700851 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:03:19.724574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.724540 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:19.724832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.724804 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.735034 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.735008 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:03:19.749119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.749085 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.749666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.749645 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:03:19.756111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.750622 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:19.763881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.756282 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.764086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.756492 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.781305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.781276 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:19.781541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.781515 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:19.789509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.789485 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:03:19.863212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.863184 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:19.864825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.864806 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:19.871476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.871445 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:03:19.872889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.872868 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:03:19.902212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.902187 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:03:19.922377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.917885 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:03:19.922377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:19.920455 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:03:20.095853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:20.095776 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:20.096198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:20.096170 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:20.097147 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:20.097127 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:03:23.324648 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.324619 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6"] Oct 30 15:03:23.329896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.329876 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.348519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.348495 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6"] Oct 30 15:03:23.358625 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.358597 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:23.358730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.358715 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:23.476378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.476334 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.476508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.476453 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.476554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.476535 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84k9j\" (UniqueName: \"kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.577889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.577818 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84k9j\" (UniqueName: \"kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.578014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.577920 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.578014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.577998 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.578644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.578624 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.580213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.580189 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.602521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.602500 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84k9j\" (UniqueName: \"kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j\") pod \"maas-api-revoke-sub-c1fd2c76-zmdc6\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.643316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.643291 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:23.955444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:23.955419 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6"] Oct 30 15:03:23.956014 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:23.955983 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3222be_241e_4297_9aae_f2b3bb95cdb9.slice/crio-3cbc642d20d588f1495c1c97d73b7e96814694f66fe81d9a4b9b0ff0f737deaa WatchSource:0}: Error finding container 3cbc642d20d588f1495c1c97d73b7e96814694f66fe81d9a4b9b0ff0f737deaa: Status 404 returned error can't find the container with id 3cbc642d20d588f1495c1c97d73b7e96814694f66fe81d9a4b9b0ff0f737deaa Oct 30 15:03:24.771642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:24.771600 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" event={"ID":"7a3222be-241e-4297-9aae-f2b3bb95cdb9","Type":"ContainerStarted","Data":"c82599470e1b96ab511215d8360398d83607afa9315bc94b4594e4f8dd5bcbea"} Oct 30 15:03:24.772126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:24.771664 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" event={"ID":"7a3222be-241e-4297-9aae-f2b3bb95cdb9","Type":"ContainerStarted","Data":"3cbc642d20d588f1495c1c97d73b7e96814694f66fe81d9a4b9b0ff0f737deaa"} Oct 30 15:03:24.819534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:24.819487 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" podStartSLOduration=1.81947378 podStartE2EDuration="1.81947378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:24.816910618 +0000 UTC m=+908.775952026" watchObservedRunningTime="2026-10-30 15:03:24.81947378 +0000 UTC m=+908.778515176" Oct 30 15:03:25.779284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:25.779245 6554 generic.go:356] "Generic (PLEG): container finished" podID="7a3222be-241e-4297-9aae-f2b3bb95cdb9" containerID="c82599470e1b96ab511215d8360398d83607afa9315bc94b4594e4f8dd5bcbea" exitCode=0 Oct 30 15:03:25.779284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:25.779281 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" event={"ID":"7a3222be-241e-4297-9aae-f2b3bb95cdb9","Type":"ContainerDied","Data":"c82599470e1b96ab511215d8360398d83607afa9315bc94b4594e4f8dd5bcbea"} Oct 30 15:03:26.942256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:26.942233 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:27.119736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.119668 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j\" (UniqueName: \"kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j\") pod \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " Oct 30 15:03:27.119736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.119730 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca\") pod \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " Oct 30 15:03:27.119917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.119857 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca\") pod \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\" (UID: \"7a3222be-241e-4297-9aae-f2b3bb95cdb9\") " Oct 30 15:03:27.120230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.120201 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca" pod "7a3222be-241e-4297-9aae-f2b3bb95cdb9" (UID: "7a3222be-241e-4297-9aae-f2b3bb95cdb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:27.121807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.121776 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca" pod "7a3222be-241e-4297-9aae-f2b3bb95cdb9" (UID: "7a3222be-241e-4297-9aae-f2b3bb95cdb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:27.121898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.121848 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j" pod "7a3222be-241e-4297-9aae-f2b3bb95cdb9" (UID: "7a3222be-241e-4297-9aae-f2b3bb95cdb9"). InnerVolumeSpecName "kube-api-access-84k9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:27.222115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.221943 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84k9j\" (UniqueName: \"kubernetes.io/projected/7a3222be-241e-4297-9aae-f2b3bb95cdb9-kube-api-access-84k9j\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:27.222115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.221972 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:27.222115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.221988 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3222be-241e-4297-9aae-f2b3bb95cdb9-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:27.792992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.792966 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" Oct 30 15:03:27.792992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.792968 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6" event={"ID":"7a3222be-241e-4297-9aae-f2b3bb95cdb9","Type":"ContainerDied","Data":"3cbc642d20d588f1495c1c97d73b7e96814694f66fe81d9a4b9b0ff0f737deaa"} Oct 30 15:03:27.793248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:27.793021 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cbc642d20d588f1495c1c97d73b7e96814694f66fe81d9a4b9b0ff0f737deaa" Oct 30 15:03:34.377502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.377442 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg"] Oct 30 15:03:34.378992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.378962 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7a3222be-241e-4297-9aae-f2b3bb95cdb9" containerName="revoke-keys" Oct 30 15:03:34.379167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.379153 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3222be-241e-4297-9aae-f2b3bb95cdb9" containerName="revoke-keys" Oct 30 15:03:34.379455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.379440 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3222be-241e-4297-9aae-f2b3bb95cdb9" containerName="revoke-keys" Oct 30 15:03:34.409497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.409468 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg"] Oct 30 15:03:34.409768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.409751 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.431798 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.431778 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:34.438173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.438150 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:34.599290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.599246 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwvv\" (UniqueName: \"kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.601899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.601873 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.602160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.602143 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.704580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.704544 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwvv\" (UniqueName: \"kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.704779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.704606 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.704779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.704645 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.706141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.706113 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.708575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.708550 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.731226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.731200 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwvv\" (UniqueName: \"kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv\") pod \"maas-api-revoke-sub-f95e2309-wccsg\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:34.760807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:34.760777 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:35.632435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:35.632385 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg"] Oct 30 15:03:36.293372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.292439 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" event={"ID":"aa05b0d6-0312-4a3b-b1f6-00d4433ae495","Type":"ContainerStarted","Data":"82d6e422899072b1a3af852fe68d10dcf19718f8c62687f90858d0562e975253"} Oct 30 15:03:36.422996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.421402 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8"] Oct 30 15:03:36.422996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.421693 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" containerID="cri-o://b7ed02841903c5f181f69b737849bca474a5ff8687e2dc1241f8a59a2df93634" gracePeriod=30 Oct 30 15:03:36.516371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.515396 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6"] Oct 30 15:03:36.516371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.515649 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" containerID="cri-o://b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544" gracePeriod=30 Oct 30 15:03:36.771454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.766621 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:36.820371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.814048 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:36.835495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.826760 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:03:36.852852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.852818 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:36.898101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.896627 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:36.932207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.898617 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:03:36.932207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:36.907704 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.051425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.006626 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.174669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.133418 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Oct 30 15:03:37.558816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.558773 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" event={"ID":"aa05b0d6-0312-4a3b-b1f6-00d4433ae495","Type":"ContainerStarted","Data":"272d484fa551661a75cbf6078a368d98f941c4202d97da54c79c15840a32e1f8"} Oct 30 15:03:37.684217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.684172 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.800377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.799452 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.800377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.800096 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:03:37.832660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.814422 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.832660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.814755 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:03:37.833378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.833333 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:03:37.833854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.833833 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:03:37.856375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.853201 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:37.856375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.853609 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:03:37.955265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.955219 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.004553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.957035 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" podStartSLOduration=3.957016571 podStartE2EDuration="3.957016571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:37.719153477 +0000 UTC m=+921.678194893" watchObservedRunningTime="2026-10-30 15:03:37.957016571 +0000 UTC m=+921.916057980" Oct 30 15:03:38.004553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.957101 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.073371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.019563 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:03:38.073371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.019643 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:03:38.073371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:37.981274 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:38.073371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.021429 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:03:38.073371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.028551 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:03:38.073371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.028850 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:03:38.237850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.237818 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:03:38.275389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.275341 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:03:38.344951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.344923 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:03:38.440289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.440247 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:03:38.551210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.548976 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:03:38.552285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.552145 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:03:38.563408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.563391 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:03:38.573374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.567423 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:03:38.586570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.586544 6554 generic.go:356] "Generic (PLEG): container finished" podID="aa05b0d6-0312-4a3b-b1f6-00d4433ae495" containerID="272d484fa551661a75cbf6078a368d98f941c4202d97da54c79c15840a32e1f8" exitCode=0 Oct 30 15:03:38.588091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:38.588057 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" event={"ID":"aa05b0d6-0312-4a3b-b1f6-00d4433ae495","Type":"ContainerDied","Data":"272d484fa551661a75cbf6078a368d98f941c4202d97da54c79c15840a32e1f8"} Oct 30 15:03:39.204813 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:03:39.204776 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:03:39.739088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.739066 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:39.837070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.837041 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca\") pod \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " Oct 30 15:03:39.837241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.837219 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca\") pod \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " Oct 30 15:03:39.837435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.837419 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv\" (UniqueName: \"kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv\") pod \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\" (UID: \"aa05b0d6-0312-4a3b-b1f6-00d4433ae495\") " Oct 30 15:03:39.837435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.837423 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca" pod "aa05b0d6-0312-4a3b-b1f6-00d4433ae495" (UID: "aa05b0d6-0312-4a3b-b1f6-00d4433ae495"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:39.837953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.837935 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:39.839581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.839554 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv" pod "aa05b0d6-0312-4a3b-b1f6-00d4433ae495" (UID: "aa05b0d6-0312-4a3b-b1f6-00d4433ae495"). InnerVolumeSpecName "kube-api-access-ddwvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:39.839686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.839597 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca" pod "aa05b0d6-0312-4a3b-b1f6-00d4433ae495" (UID: "aa05b0d6-0312-4a3b-b1f6-00d4433ae495"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:39.939191 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.939163 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwvv\" (UniqueName: \"kubernetes.io/projected/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-kube-api-access-ddwvv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:39.939191 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.939191 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa05b0d6-0312-4a3b-b1f6-00d4433ae495-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:39.997803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.997777 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j"] Oct 30 15:03:39.998373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.998343 6554 cpu_manager.go:414] "containerMap: removing container" podUID="aa05b0d6-0312-4a3b-b1f6-00d4433ae495" containerName="revoke-keys" Oct 30 15:03:39.998424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.998375 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa05b0d6-0312-4a3b-b1f6-00d4433ae495" containerName="revoke-keys" Oct 30 15:03:39.998508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:39.998497 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa05b0d6-0312-4a3b-b1f6-00d4433ae495" containerName="revoke-keys" Oct 30 15:03:40.002000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.001982 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.030995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.030971 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j"] Oct 30 15:03:40.040725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.040700 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.040858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.040736 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spf49\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.040858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.040758 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.040986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.040904 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.040986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.040964 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.041090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.041050 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.041142 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.041131 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.041194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.041169 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.041251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.041230 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142224 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142258 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spf49\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142279 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142316 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142389 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142436 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142509 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142636 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142688 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142863 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142922 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.142992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.142949 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.143064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.143045 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.143100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.143066 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.144830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.144813 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.144902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.144820 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.166793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.166770 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spf49\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.166879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.166864 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.317903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.317884 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:40.601335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.601272 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" Oct 30 15:03:40.601489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.601277 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg" event={"ID":"aa05b0d6-0312-4a3b-b1f6-00d4433ae495","Type":"ContainerDied","Data":"82d6e422899072b1a3af852fe68d10dcf19718f8c62687f90858d0562e975253"} Oct 30 15:03:40.601489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.601396 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d6e422899072b1a3af852fe68d10dcf19718f8c62687f90858d0562e975253" Oct 30 15:03:40.638876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.638853 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j"] Oct 30 15:03:40.639516 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:40.639481 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52292a7b_4a45_4110_af04_a35cbb09893f.slice/crio-42ddd2993f0a35e1c5f16e63236e5a3021f4416eea09a4b6304e717dba60b331 WatchSource:0}: Error finding container 42ddd2993f0a35e1c5f16e63236e5a3021f4416eea09a4b6304e717dba60b331: Status 404 returned error can't find the container with id 42ddd2993f0a35e1c5f16e63236e5a3021f4416eea09a4b6304e717dba60b331 Oct 30 15:03:40.641891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.641861 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:40.641979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.641921 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:40.641979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:40.641950 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:41.612682 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.612653 6554 generic.go:356] "Generic (PLEG): container finished" podID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerID="b7ed02841903c5f181f69b737849bca474a5ff8687e2dc1241f8a59a2df93634" exitCode=0 Oct 30 15:03:41.613154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.612747 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" event={"ID":"996e7f9d-6aee-4621-9f6b-10892c43f243","Type":"ContainerDied","Data":"b7ed02841903c5f181f69b737849bca474a5ff8687e2dc1241f8a59a2df93634"} Oct 30 15:03:41.614930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.614899 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" event={"ID":"52292a7b-4a45-4110-af04-a35cbb09893f","Type":"ContainerStarted","Data":"e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1"} Oct 30 15:03:41.615036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.614953 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" event={"ID":"52292a7b-4a45-4110-af04-a35cbb09893f","Type":"ContainerStarted","Data":"42ddd2993f0a35e1c5f16e63236e5a3021f4416eea09a4b6304e717dba60b331"} Oct 30 15:03:41.682203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.682148 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podStartSLOduration=2.682131794 podStartE2EDuration="2.682131794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:41.679908404 +0000 UTC m=+925.638949818" watchObservedRunningTime="2026-10-30 15:03:41.682131794 +0000 UTC m=+925.641173198" Oct 30 15:03:41.802725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.802545 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 15:03:41.893271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893244 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893282 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893321 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893406 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893453 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893503 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893607 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.893686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893649 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893673 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.893720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893695 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.893806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893789 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.893840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.893824 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert\") pod \"996e7f9d-6aee-4621-9f6b-10892c43f243\" (UID: \"996e7f9d-6aee-4621-9f6b-10892c43f243\") " Oct 30 15:03:41.894145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894107 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.894418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894395 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:41.894544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894528 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996e7f9d-6aee-4621-9f6b-10892c43f243-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.894614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894547 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.894614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894569 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.894614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894594 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.894614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.894602 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.896711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.896683 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.896809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.896793 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:41.897728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.897708 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:03:41.898209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.898192 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5" pod "996e7f9d-6aee-4621-9f6b-10892c43f243" (UID: "996e7f9d-6aee-4621-9f6b-10892c43f243"). InnerVolumeSpecName "kube-api-access-zl2k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:41.902541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.902525 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995613 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995661 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995709 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995754 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995890 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995938 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.995986 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996008 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996113 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket\") pod \"0b5a9032-6fb8-4833-992c-e128a4419f73\" (UID: \"0b5a9032-6fb8-4833-992c-e128a4419f73\") " Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996610 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996624 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl2k5\" (UniqueName: \"kubernetes.io/projected/996e7f9d-6aee-4621-9f6b-10892c43f243-kube-api-access-zl2k5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996633 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996642 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996e7f9d-6aee-4621-9f6b-10892c43f243-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:41.996831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.996820 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.997471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.997028 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.997471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.997367 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:41.997471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.997442 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.998724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.998695 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:41.999994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:41.999968 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:03:42.000529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.000506 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:42.001463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.001442 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:03:42.002118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.002098 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l" pod "0b5a9032-6fb8-4833-992c-e128a4419f73" (UID: "0b5a9032-6fb8-4833-992c-e128a4419f73"). InnerVolumeSpecName "kube-api-access-tjg2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097649 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097679 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b5a9032-6fb8-4833-992c-e128a4419f73-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097689 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097699 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097707 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097715 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097724 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjg2l\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-kube-api-access-tjg2l\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.097731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097734 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5a9032-6fb8-4833-992c-e128a4419f73-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.098121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.097744 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b5a9032-6fb8-4833-992c-e128a4419f73-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:42.318474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.318446 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:42.320237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.320212 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 15:03:42.320367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.320258 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 15:03:42.622222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.622192 6554 generic.go:356] "Generic (PLEG): container finished" podID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerID="b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544" exitCode=0 Oct 30 15:03:42.622721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.622251 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" Oct 30 15:03:42.622721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.622287 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" event={"ID":"0b5a9032-6fb8-4833-992c-e128a4419f73","Type":"ContainerDied","Data":"b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544"} Oct 30 15:03:42.622721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.622333 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6" event={"ID":"0b5a9032-6fb8-4833-992c-e128a4419f73","Type":"ContainerDied","Data":"da320cc10e96c7313e0df731291aa7e6f12376b73514c46c2acfe965c6b8340e"} Oct 30 15:03:42.622721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.622377 6554 scope.go:122] "RemoveContainer" containerID="b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544" Oct 30 15:03:42.623857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.623831 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" Oct 30 15:03:42.623971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.623881 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8" event={"ID":"996e7f9d-6aee-4621-9f6b-10892c43f243","Type":"ContainerDied","Data":"586cca9aa35ec6aaf603c024759ed7594a458089c8720105cad4d44b34ab298f"} Oct 30 15:03:42.632770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.632753 6554 scope.go:122] "RemoveContainer" containerID="b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544" Oct 30 15:03:42.633020 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:03:42.632995 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544\": container with ID starting with b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544 not found: ID does not exist" containerID="b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544" Oct 30 15:03:42.633122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.633022 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544"} err="failed to get container status \"b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544\": rpc error: code = NotFound desc = could not find container \"b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544\": container with ID starting with b988ff166a4d351b219e6bc6880bac701a4695d356df91ac7bfa644abe7bd544 not found: ID does not exist" Oct 30 15:03:42.633122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.633040 6554 scope.go:122] "RemoveContainer" containerID="b7ed02841903c5f181f69b737849bca474a5ff8687e2dc1241f8a59a2df93634" Oct 30 15:03:42.700056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.700033 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8"] Oct 30 15:03:42.704840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.704814 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-tw6z8"] Oct 30 15:03:42.742381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.742361 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6"] Oct 30 15:03:42.747977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:42.747956 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-bh8r6"] Oct 30 15:03:43.318295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:43.318261 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 15:03:43.318495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:43.318329 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 15:03:44.177372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:44.177312 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" path="/var/lib/kubelet/pods/0b5a9032-6fb8-4833-992c-e128a4419f73/volumes" Oct 30 15:03:44.178462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:44.178042 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" path="/var/lib/kubelet/pods/996e7f9d-6aee-4621-9f6b-10892c43f243/volumes" Oct 30 15:03:44.318688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:44.318654 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 15:03:44.318841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:44.318707 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 15:03:45.318757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:45.318723 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 15:03:45.319248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:45.318770 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 15:03:46.319169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.319136 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 15:03:46.319582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.319194 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 15:03:46.993690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.993655 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg"] Oct 30 15:03:46.994231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.994219 6554 cpu_manager.go:414] "containerMap: removing container" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" Oct 30 15:03:46.994286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.994232 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" Oct 30 15:03:46.994286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.994253 6554 cpu_manager.go:414] "containerMap: removing container" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" Oct 30 15:03:46.994286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.994257 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" Oct 30 15:03:46.994414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.994342 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5a9032-6fb8-4833-992c-e128a4419f73" containerName="istio-proxy" Oct 30 15:03:46.994414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.994372 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="996e7f9d-6aee-4621-9f6b-10892c43f243" containerName="istio-proxy" Oct 30 15:03:46.998943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:46.998927 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.024886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.024855 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg"] Oct 30 15:03:47.027529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.027507 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:03:47.027687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.027508 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:03:47.162180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.162147 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.162180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.162186 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v56j\" (UniqueName: \"kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.162400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.162325 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.263709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.263629 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.263709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.263680 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v56j\" (UniqueName: \"kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.264240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.264203 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.264423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.264396 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.266533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.266506 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.289008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.288983 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v56j\" (UniqueName: \"kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j\") pod \"maas-api-revoke-sub-5547ad9f-mz8xg\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.318235 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.318211 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 15:03:47.318328 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.318257 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 15:03:47.329969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.329951 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:47.666773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:47.666743 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg"] Oct 30 15:03:48.323111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:48.323041 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:48.323507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:48.323478 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:48.324280 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:48.324260 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:03:48.667250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:48.667214 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" event={"ID":"ee5f40f7-63e1-45ad-b776-57374f833d59","Type":"ContainerStarted","Data":"905fb5bae65774419a29b7cc7c2aa0e08134fac50374323f2b638ffbc6d5398c"} Oct 30 15:03:48.667675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:48.667275 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" event={"ID":"ee5f40f7-63e1-45ad-b776-57374f833d59","Type":"ContainerStarted","Data":"ad51275fe57a2ff3cce50c7d233b38904e160df29cc626a6528afa47459a6baf"} Oct 30 15:03:48.710044 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:48.709990 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" podStartSLOduration=2.7099755229999998 podStartE2EDuration="2.709975523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:48.709652249 +0000 UTC m=+932.668693659" watchObservedRunningTime="2026-10-30 15:03:48.709975523 +0000 UTC m=+932.669016927" Oct 30 15:03:49.679153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:49.679101 6554 generic.go:356] "Generic (PLEG): container finished" podID="ee5f40f7-63e1-45ad-b776-57374f833d59" containerID="905fb5bae65774419a29b7cc7c2aa0e08134fac50374323f2b638ffbc6d5398c" exitCode=0 Oct 30 15:03:49.679609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:49.679170 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" event={"ID":"ee5f40f7-63e1-45ad-b776-57374f833d59","Type":"ContainerDied","Data":"905fb5bae65774419a29b7cc7c2aa0e08134fac50374323f2b638ffbc6d5398c"} Oct 30 15:03:50.925469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:50.925431 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:50.926058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:50.926026 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.190545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.190505 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:03:51.191697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.191665 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.216441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.214577 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.285483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.285437 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.304483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.304433 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.370743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.370704 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.563437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.542868 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.585537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.585489 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.645376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.612959 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:03:51.649373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.646058 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.679942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.649308 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 15:03:51.680484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.671077 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:03:51.913008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.912968 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:51.958048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.958011 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" event={"ID":"ee5f40f7-63e1-45ad-b776-57374f833d59","Type":"ContainerDied","Data":"ad51275fe57a2ff3cce50c7d233b38904e160df29cc626a6528afa47459a6baf"} Oct 30 15:03:51.978264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:51.978236 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad51275fe57a2ff3cce50c7d233b38904e160df29cc626a6528afa47459a6baf" Oct 30 15:03:52.015692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.015662 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:52.209434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.189035 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca\") pod \"ee5f40f7-63e1-45ad-b776-57374f833d59\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " Oct 30 15:03:52.229955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.219427 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.236257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.229860 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca" pod "ee5f40f7-63e1-45ad-b776-57374f833d59" (UID: "ee5f40f7-63e1-45ad-b776-57374f833d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:03:52.236442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.235662 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca\") pod \"ee5f40f7-63e1-45ad-b776-57374f833d59\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " Oct 30 15:03:52.260956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.260926 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j\" (UniqueName: \"kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j\") pod \"ee5f40f7-63e1-45ad-b776-57374f833d59\" (UID: \"ee5f40f7-63e1-45ad-b776-57374f833d59\") " Oct 30 15:03:52.288468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.288425 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.395192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.334716 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:03:52.395192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.334780 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:03:52.395192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.337138 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:52.395192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.371462 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.395192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.372081 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:03:52.477598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.477510 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca" pod "ee5f40f7-63e1-45ad-b776-57374f833d59" (UID: "ee5f40f7-63e1-45ad-b776-57374f833d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:03:52.482537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.482505 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j" pod "ee5f40f7-63e1-45ad-b776-57374f833d59" (UID: "ee5f40f7-63e1-45ad-b776-57374f833d59"). InnerVolumeSpecName "kube-api-access-5v56j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:03:52.565500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.565451 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:03:52.566443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.566417 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:03:52.592261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.568109 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v56j\" (UniqueName: \"kubernetes.io/projected/ee5f40f7-63e1-45ad-b776-57374f833d59-kube-api-access-5v56j\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:52.596435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.596404 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee5f40f7-63e1-45ad-b776-57374f833d59-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:03:52.625370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.622076 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.691441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.684958 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:03:52.691441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.649484 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.691441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.680564 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.701748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.680522 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.718384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.712233 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:03:52.718384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.712301 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:03:52.718384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.712340 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:03:52.775425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.772485 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:52.775425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:52.773012 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:03:53.054449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.050761 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg" Oct 30 15:03:53.306424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.293011 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:03:53.323467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.323418 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.372658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.372605 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.508874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.508820 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.568528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.568427 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:03:53.688757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.688712 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.689292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.689260 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 15:03:53.689725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.689699 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:03:53.716553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.714282 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 15:03:53.717387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.717338 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:03:53.914398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.911170 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:03:53.914398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.912396 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:03:53.959374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.949657 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:03:53.959374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.957568 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:03:54.003022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:53.974844 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:03:54.107447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.104772 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg"] Oct 30 15:03:54.115416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.115328 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ee5f40f7-63e1-45ad-b776-57374f833d59" containerName="revoke-keys" Oct 30 15:03:54.115917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.115897 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5f40f7-63e1-45ad-b776-57374f833d59" containerName="revoke-keys" Oct 30 15:03:54.116515 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.116495 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5f40f7-63e1-45ad-b776-57374f833d59" containerName="revoke-keys" Oct 30 15:03:54.137761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.137735 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.144082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.144056 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:03:54.152467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.152446 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg"] Oct 30 15:03:54.191658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.189380 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:03:54.193101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.192305 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:03:54.299381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.299329 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.299563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.299390 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.299563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.299425 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.299563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.299450 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.299563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.299530 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.300263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.300234 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf65s\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.300419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.300369 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.300419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.300408 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.300645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.300610 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.323759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.323736 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh"] Oct 30 15:03:54.327969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.327952 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.357753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.357723 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh"] Oct 30 15:03:54.401939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.401902 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.402104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.401952 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.402104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402039 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvtmz\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.402229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402152 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.402289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402235 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf65s\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.402362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402290 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.402362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402323 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.402476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402366 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.402476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402375 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.402607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402503 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.402698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402675 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.402855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402830 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.402945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402928 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.403009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402965 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.403009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402993 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.403117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.403031 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.403117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.403056 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.403117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.403093 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.403261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.403124 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.403467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.403333 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.403555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.402719 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.403748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.403715 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.404069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.404046 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.405137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.405113 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.406495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.406472 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.427290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.427266 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf65s\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.428618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.428595 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token\") pod \"e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.504796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504735 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.504891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504816 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.504891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504855 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.504958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504918 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.504958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504941 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504967 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.504996 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505040 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvtmz\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505091 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505297 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505469 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505651 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505729 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505701 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.505729 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.505718 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.507180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.507160 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.507324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.507307 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.516280 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.516257 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:54.529672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.529652 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.529829 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.529814 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvtmz\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz\") pod \"e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.644144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.644110 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:54.840425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.840396 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg"] Oct 30 15:03:54.840577 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:54.840541 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f83f289_4683_4f0a_a85a_41045ade9be9.slice/crio-7ec10a1c1dff4b91da49db48df71835d7fdc23aebc73f3f6f60a28cc65f8df5d WatchSource:0}: Error finding container 7ec10a1c1dff4b91da49db48df71835d7fdc23aebc73f3f6f60a28cc65f8df5d: Status 404 returned error can't find the container with id 7ec10a1c1dff4b91da49db48df71835d7fdc23aebc73f3f6f60a28cc65f8df5d Oct 30 15:03:54.842982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.842944 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:54.843085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.843012 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:54.843085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.843047 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:54.968988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.968959 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh"] Oct 30 15:03:54.969931 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:03:54.969905 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a2cff1_a2e4_4c9d_a474_279529d4a336.slice/crio-24f169781c4f779db782fced3036a7c4a40c2f7beb7a916588f833d48b599c22 WatchSource:0}: Error finding container 24f169781c4f779db782fced3036a7c4a40c2f7beb7a916588f833d48b599c22: Status 404 returned error can't find the container with id 24f169781c4f779db782fced3036a7c4a40c2f7beb7a916588f833d48b599c22 Oct 30 15:03:54.972780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.972733 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:54.972886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.972817 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:54.972886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:54.972854 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:55.032316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.032288 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc"] Oct 30 15:03:55.037801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.037779 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.063300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.063269 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc"] Oct 30 15:03:55.115916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.115822 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.115916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.115903 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116029 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db75v\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116084 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116126 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116202 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116226 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116250 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.116339 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.131955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.131921 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" event={"ID":"f7a2cff1-a2e4-4c9d-a474-279529d4a336","Type":"ContainerStarted","Data":"6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4"} Oct 30 15:03:55.132126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.131975 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" event={"ID":"f7a2cff1-a2e4-4c9d-a474-279529d4a336","Type":"ContainerStarted","Data":"24f169781c4f779db782fced3036a7c4a40c2f7beb7a916588f833d48b599c22"} Oct 30 15:03:55.134179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.134152 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" event={"ID":"3f83f289-4683-4f0a-a85a-41045ade9be9","Type":"ContainerStarted","Data":"52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51"} Oct 30 15:03:55.134303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.134204 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" event={"ID":"3f83f289-4683-4f0a-a85a-41045ade9be9","Type":"ContainerStarted","Data":"7ec10a1c1dff4b91da49db48df71835d7fdc23aebc73f3f6f60a28cc65f8df5d"} Oct 30 15:03:55.182114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.182062 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podStartSLOduration=1.1820492360000001 podStartE2EDuration="1.182049236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:55.181508549 +0000 UTC m=+939.140549954" watchObservedRunningTime="2026-10-30 15:03:55.182049236 +0000 UTC m=+939.141090640" Oct 30 15:03:55.217795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.217748 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.217960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.217927 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.218035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.217975 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db75v\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.218035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218012 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.218144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218089 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218257 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218257 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218420 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218497 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218561 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.218736 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.219207 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.219226 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.219441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.219283 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.220659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.220622 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.221420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.221399 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.230930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.230872 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podStartSLOduration=2.230857017 podStartE2EDuration="2.230857017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:55.229184841 +0000 UTC m=+939.188226247" watchObservedRunningTime="2026-10-30 15:03:55.230857017 +0000 UTC m=+939.189898427" Oct 30 15:03:55.247965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.247939 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.248291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.248270 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db75v\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v\") pod \"e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.354641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.354594 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:55.521608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.521520 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:03:55.524602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.524562 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:55.524791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.524627 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:03:55.644982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.644954 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:03:55.703409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.699392 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc"] Oct 30 15:03:55.720333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.720301 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:55.720426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.720393 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:55.720485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:55.720434 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:03:56.144300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.144260 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" event={"ID":"9e5a0723-eb84-458e-a15e-180c90d45e37","Type":"ContainerStarted","Data":"328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5"} Oct 30 15:03:56.144830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.144324 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" event={"ID":"9e5a0723-eb84-458e-a15e-180c90d45e37","Type":"ContainerStarted","Data":"8b2c176dc2d241c3c7a4b7c5dae7f90c96d43cb36e81f78b5a477b65355f9b26"} Oct 30 15:03:56.221990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.216656 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podStartSLOduration=1.216637326 podStartE2EDuration="1.216637326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:03:56.213131752 +0000 UTC m=+940.172173160" watchObservedRunningTime="2026-10-30 15:03:56.216637326 +0000 UTC m=+940.175678734" Oct 30 15:03:56.374369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.368428 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:03:56.374369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.368558 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:03:56.374369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.368622 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:03:56.517008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.516910 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:56.517008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.516970 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:03:56.648374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.645871 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:56.648374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:56.645965 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 15:03:57.358377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:57.357777 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:03:57.358377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:57.357851 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:03:57.520814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:57.520775 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:57.521073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:57.521047 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:03:57.644894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:57.644863 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:57.645061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:57.644913 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 15:03:58.359565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:58.357711 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:03:58.359565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:58.357782 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:03:58.524373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:58.518678 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:58.524373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:58.518752 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:03:58.723374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:58.721116 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:58.723374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:58.721176 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 15:03:59.361374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:59.357731 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:03:59.361374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:59.357792 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:03:59.522382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:59.521807 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:03:59.522382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:59.521872 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:03:59.644938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:59.644910 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:03:59.645093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:03:59.644969 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 15:04:00.362837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:00.362800 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:04:00.363472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:00.363426 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:04:00.528832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:00.528795 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:04:00.529100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:00.529068 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:04:00.645069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:00.645040 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:04:00.645283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:00.645262 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 15:04:01.358379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:01.357715 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:04:01.358379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:01.357776 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:04:01.517393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:01.517339 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 15:04:01.517785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:01.517412 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 15:04:01.645165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:01.645131 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 15:04:01.645335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:01.645195 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 15:04:02.355594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.355553 6554 patch_prober.go:29] interesting pod/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 15:04:02.355803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.355620 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 15:04:02.594872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.594840 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:04:02.595587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.595561 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:04:02.596526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.596504 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:04:02.826884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.826833 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:04:02.827542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.827489 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:04:02.829993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:02.829971 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:04:03.776429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:03.776393 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:04:03.777601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:03.777584 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:04:03.778293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:03.778274 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:04:05.517229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.517194 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45"] Oct 30 15:04:05.521674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.521658 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.550953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.550921 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45"] Oct 30 15:04:05.554522 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.554494 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.554643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.554619 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztlj6\" (UniqueName: \"kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.554711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.554703 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.560550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.560529 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:05.560662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.560567 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:05.655529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.655489 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.655674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.655601 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztlj6\" (UniqueName: \"kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.655751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.655708 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.656340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.656314 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.657835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.657814 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.680498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.680473 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztlj6\" (UniqueName: \"kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6\") pod \"maas-api-revoke-sub-2e6bd936-b9r45\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:05.835756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:05.835692 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:06.138296 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:04:06.138272 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36a6b811_11d0_4712_b551_37a74a2b96a3.slice/crio-0d8642dc8f3e87dfceb50d8131b29d360310b90b375409c126bf41a4eeb601ab WatchSource:0}: Error finding container 0d8642dc8f3e87dfceb50d8131b29d360310b90b375409c126bf41a4eeb601ab: Status 404 returned error can't find the container with id 0d8642dc8f3e87dfceb50d8131b29d360310b90b375409c126bf41a4eeb601ab Oct 30 15:04:06.138741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:06.138718 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45"] Oct 30 15:04:06.293339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:06.293306 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" event={"ID":"36a6b811-11d0-4712-b551-37a74a2b96a3","Type":"ContainerStarted","Data":"0d8642dc8f3e87dfceb50d8131b29d360310b90b375409c126bf41a4eeb601ab"} Oct 30 15:04:07.304506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:07.304431 6554 generic.go:356] "Generic (PLEG): container finished" podID="36a6b811-11d0-4712-b551-37a74a2b96a3" containerID="5a2b7e4b884f1c2f51943a4348d7ffaec0a28cb7bb611d4188788e8fe9149305" exitCode=0 Oct 30 15:04:07.304506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:07.304496 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" event={"ID":"36a6b811-11d0-4712-b551-37a74a2b96a3","Type":"ContainerDied","Data":"5a2b7e4b884f1c2f51943a4348d7ffaec0a28cb7bb611d4188788e8fe9149305"} Oct 30 15:04:09.291375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.274132 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.305372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.302708 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.559667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.556458 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.602865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.602431 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:04:09.807373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.750540 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 15:04:09.807373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.750905 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:04:09.807373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.786721 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:04:09.827419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.818798 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.861374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.828976 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:09.916097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.915783 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca\") pod \"36a6b811-11d0-4712-b551-37a74a2b96a3\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " Oct 30 15:04:09.938195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.916748 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:09.938195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.927216 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6\" (UniqueName: \"kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6\") pod \"36a6b811-11d0-4712-b551-37a74a2b96a3\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " Oct 30 15:04:09.938195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.927273 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca\") pod \"36a6b811-11d0-4712-b551-37a74a2b96a3\" (UID: \"36a6b811-11d0-4712-b551-37a74a2b96a3\") " Oct 30 15:04:09.938195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.928293 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca" pod "36a6b811-11d0-4712-b551-37a74a2b96a3" (UID: "36a6b811-11d0-4712-b551-37a74a2b96a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:09.969374 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:04:09.963598 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:04:09.981561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.981503 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca" pod "36a6b811-11d0-4712-b551-37a74a2b96a3" (UID: "36a6b811-11d0-4712-b551-37a74a2b96a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:09.984839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.984792 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6" pod "36a6b811-11d0-4712-b551-37a74a2b96a3" (UID: "36a6b811-11d0-4712-b551-37a74a2b96a3"). InnerVolumeSpecName "kube-api-access-ztlj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:10.018423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:09.990445 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.035368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.034130 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztlj6\" (UniqueName: \"kubernetes.io/projected/36a6b811-11d0-4712-b551-37a74a2b96a3-kube-api-access-ztlj6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:10.035368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.034165 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:10.035368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.034180 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36a6b811-11d0-4712-b551-37a74a2b96a3-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:10.327598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.327550 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:04:10.332118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.331413 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.407379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.386999 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:04:10.407379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.387062 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:04:10.561785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.558576 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.625449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.583009 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:04:10.631390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.626578 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:10.631390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.627000 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:04:10.753196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.753162 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" event={"ID":"36a6b811-11d0-4712-b551-37a74a2b96a3","Type":"ContainerDied","Data":"0d8642dc8f3e87dfceb50d8131b29d360310b90b375409c126bf41a4eeb601ab"} Oct 30 15:04:10.758630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.758510 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8642dc8f3e87dfceb50d8131b29d360310b90b375409c126bf41a4eeb601ab" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.758963 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.795461 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.795501 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.796502 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.925484 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:10.976244 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.039374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.035374 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:04:11.056974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.056934 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:04:11.057315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.057296 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:04:11.057703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.057687 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:04:11.057914 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.057884 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:04:11.058088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.058068 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:04:11.115374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.082599 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:04:11.164391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.164343 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:04:11.347703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.345472 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.347703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.345871 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:11.408093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.393929 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:04:11.474379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.454829 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:04:11.555383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.547865 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:04:11.569381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.567852 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:04:11.569381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.567913 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:04:11.591373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.581760 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:04:11.591615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.591598 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:04:11.609827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:11.607696 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:04:13.286181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.286145 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97"] Oct 30 15:04:13.287052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.287034 6554 cpu_manager.go:414] "containerMap: removing container" podUID="36a6b811-11d0-4712-b551-37a74a2b96a3" containerName="revoke-keys" Oct 30 15:04:13.287052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.287054 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36a6b811-11d0-4712-b551-37a74a2b96a3" containerName="revoke-keys" Oct 30 15:04:13.287193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.287180 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="36a6b811-11d0-4712-b551-37a74a2b96a3" containerName="revoke-keys" Oct 30 15:04:13.291821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.291719 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.311035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.310996 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97"] Oct 30 15:04:13.311878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.311548 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:13.319775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.319753 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:13.411648 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.411609 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.412106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.412079 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vdhc\" (UniqueName: \"kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.412409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.412343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.513689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.513662 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.513800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.513742 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vdhc\" (UniqueName: \"kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.513800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.513781 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.514437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.514415 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.516245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.516223 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.538696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.538648 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vdhc\" (UniqueName: \"kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc\") pod \"maas-api-revoke-sub-0f8f51ca-tfb97\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.620531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.620511 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:13.923205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:13.923176 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97"] Oct 30 15:04:13.924197 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:04:13.924173 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3400b3d8_d4fe_4e35_a06b_d1d9147c5b1f.slice/crio-ed4de7b57b29fc003a319b0cc2dcb67c82dcd4738612731c6f7aa801acfff1b3 WatchSource:0}: Error finding container ed4de7b57b29fc003a319b0cc2dcb67c82dcd4738612731c6f7aa801acfff1b3: Status 404 returned error can't find the container with id ed4de7b57b29fc003a319b0cc2dcb67c82dcd4738612731c6f7aa801acfff1b3 Oct 30 15:04:14.785552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:14.785519 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" event={"ID":"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f","Type":"ContainerStarted","Data":"ada98a45ba987edc837beb32af20c71d9d7098e7675e8c58d409409df49617ea"} Oct 30 15:04:14.786000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:14.785584 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" event={"ID":"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f","Type":"ContainerStarted","Data":"ed4de7b57b29fc003a319b0cc2dcb67c82dcd4738612731c6f7aa801acfff1b3"} Oct 30 15:04:14.832134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:14.832091 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" podStartSLOduration=1.832078047 podStartE2EDuration="1.832078047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:14.829277995 +0000 UTC m=+958.788319400" watchObservedRunningTime="2026-10-30 15:04:14.832078047 +0000 UTC m=+958.791119452" Oct 30 15:04:15.793873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:15.793840 6554 generic.go:356] "Generic (PLEG): container finished" podID="3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" containerID="ada98a45ba987edc837beb32af20c71d9d7098e7675e8c58d409409df49617ea" exitCode=0 Oct 30 15:04:15.794267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:15.793904 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" event={"ID":"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f","Type":"ContainerDied","Data":"ada98a45ba987edc837beb32af20c71d9d7098e7675e8c58d409409df49617ea"} Oct 30 15:04:16.956605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:16.956582 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:17.055255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.055214 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc\" (UniqueName: \"kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc\") pod \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " Oct 30 15:04:17.055548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.055524 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca\") pod \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " Oct 30 15:04:17.055634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.055590 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca\") pod \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\" (UID: \"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f\") " Oct 30 15:04:17.055956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.055925 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca" pod "3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" (UID: "3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:17.056296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.056274 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:17.058197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.058166 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc" pod "3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" (UID: "3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f"). InnerVolumeSpecName "kube-api-access-6vdhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:17.059763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.059728 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca" pod "3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" (UID: "3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:17.158249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.158211 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:17.158249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.158248 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vdhc\" (UniqueName: \"kubernetes.io/projected/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f-kube-api-access-6vdhc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:17.808944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.808908 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" event={"ID":"3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f","Type":"ContainerDied","Data":"ed4de7b57b29fc003a319b0cc2dcb67c82dcd4738612731c6f7aa801acfff1b3"} Oct 30 15:04:17.809132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.808949 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97" Oct 30 15:04:17.809132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:17.808980 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed4de7b57b29fc003a319b0cc2dcb67c82dcd4738612731c6f7aa801acfff1b3" Oct 30 15:04:22.488957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.488906 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9"] Oct 30 15:04:22.490703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.490677 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" containerName="revoke-keys" Oct 30 15:04:22.490882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.490869 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" containerName="revoke-keys" Oct 30 15:04:22.491148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.491135 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" containerName="revoke-keys" Oct 30 15:04:22.495706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.495682 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.511633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.511608 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9"] Oct 30 15:04:22.521393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.521338 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:22.521620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.521595 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:22.618089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.618067 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.618238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.618218 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.618395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.618380 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m69j8\" (UniqueName: \"kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.719231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.719209 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m69j8\" (UniqueName: \"kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.719343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.719261 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.719343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.719331 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.719904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.719881 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.721630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.721605 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.744458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.744404 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m69j8\" (UniqueName: \"kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8\") pod \"maas-api-revoke-sub-a6f0a7cb-jq5k9\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:22.818722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:22.810471 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:23.129493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:23.129467 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9"] Oct 30 15:04:23.130396 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:04:23.130339 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd626d53_f2d5_472e_8ab4_aca0a8e8923d.slice/crio-113b76cb8ca762a615ddfc978a6fc14c82d024ed37b134cdc65e0b947380673e WatchSource:0}: Error finding container 113b76cb8ca762a615ddfc978a6fc14c82d024ed37b134cdc65e0b947380673e: Status 404 returned error can't find the container with id 113b76cb8ca762a615ddfc978a6fc14c82d024ed37b134cdc65e0b947380673e Oct 30 15:04:23.850311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:23.850234 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" event={"ID":"bd626d53-f2d5-472e-8ab4-aca0a8e8923d","Type":"ContainerStarted","Data":"02ce499991908b18696c27183e06bd129c155b250798c5dcfa155a8ae0f0f616"} Oct 30 15:04:23.850311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:23.850299 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" event={"ID":"bd626d53-f2d5-472e-8ab4-aca0a8e8923d","Type":"ContainerStarted","Data":"113b76cb8ca762a615ddfc978a6fc14c82d024ed37b134cdc65e0b947380673e"} Oct 30 15:04:23.893445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:23.893397 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" podStartSLOduration=1.893379986 podStartE2EDuration="1.893379986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:23.89281052 +0000 UTC m=+967.851851924" watchObservedRunningTime="2026-10-30 15:04:23.893379986 +0000 UTC m=+967.852421391" Oct 30 15:04:24.858186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:24.858152 6554 generic.go:356] "Generic (PLEG): container finished" podID="bd626d53-f2d5-472e-8ab4-aca0a8e8923d" containerID="02ce499991908b18696c27183e06bd129c155b250798c5dcfa155a8ae0f0f616" exitCode=0 Oct 30 15:04:24.858593 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:24.858206 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" event={"ID":"bd626d53-f2d5-472e-8ab4-aca0a8e8923d","Type":"ContainerDied","Data":"02ce499991908b18696c27183e06bd129c155b250798c5dcfa155a8ae0f0f616"} Oct 30 15:04:26.013023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.013001 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:26.068677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.068649 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8\" (UniqueName: \"kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8\") pod \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " Oct 30 15:04:26.068820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.068777 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca\") pod \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " Oct 30 15:04:26.069018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.068998 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca\") pod \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\" (UID: \"bd626d53-f2d5-472e-8ab4-aca0a8e8923d\") " Oct 30 15:04:26.069169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.069148 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca" pod "bd626d53-f2d5-472e-8ab4-aca0a8e8923d" (UID: "bd626d53-f2d5-472e-8ab4-aca0a8e8923d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:26.069563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.069540 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:26.071085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.071056 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca" pod "bd626d53-f2d5-472e-8ab4-aca0a8e8923d" (UID: "bd626d53-f2d5-472e-8ab4-aca0a8e8923d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:26.071317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.071295 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8" pod "bd626d53-f2d5-472e-8ab4-aca0a8e8923d" (UID: "bd626d53-f2d5-472e-8ab4-aca0a8e8923d"). InnerVolumeSpecName "kube-api-access-m69j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:26.170938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.170909 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m69j8\" (UniqueName: \"kubernetes.io/projected/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-kube-api-access-m69j8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:26.170938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.170938 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd626d53-f2d5-472e-8ab4-aca0a8e8923d-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:26.872871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.872847 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" Oct 30 15:04:26.872871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.872856 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9" event={"ID":"bd626d53-f2d5-472e-8ab4-aca0a8e8923d","Type":"ContainerDied","Data":"113b76cb8ca762a615ddfc978a6fc14c82d024ed37b134cdc65e0b947380673e"} Oct 30 15:04:26.873066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.872899 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="113b76cb8ca762a615ddfc978a6fc14c82d024ed37b134cdc65e0b947380673e" Oct 30 15:04:26.953551 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.953518 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx"] Oct 30 15:04:26.954170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.954149 6554 cpu_manager.go:414] "containerMap: removing container" podUID="bd626d53-f2d5-472e-8ab4-aca0a8e8923d" containerName="revoke-keys" Oct 30 15:04:26.954170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.954170 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd626d53-f2d5-472e-8ab4-aca0a8e8923d" containerName="revoke-keys" Oct 30 15:04:26.954367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.954338 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd626d53-f2d5-472e-8ab4-aca0a8e8923d" containerName="revoke-keys" Oct 30 15:04:26.958050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.958030 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981432 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981548 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981622 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4mc\" (UniqueName: \"kubernetes.io/projected/3c8367b1-00aa-40d7-ab02-e25379a096c0-kube-api-access-5n4mc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981662 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981729 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981773 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8367b1-00aa-40d7-ab02-e25379a096c0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.981824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.981817 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:26.990962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.990440 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 15:04:26.990962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.990454 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 15:04:26.990962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.990481 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 15:04:26.990962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.990518 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jz5vp\"" Oct 30 15:04:26.993835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:26.993815 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx"] Oct 30 15:04:27.082713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.082682 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4mc\" (UniqueName: \"kubernetes.io/projected/3c8367b1-00aa-40d7-ab02-e25379a096c0-kube-api-access-5n4mc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.082729 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.082777 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.082832 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8367b1-00aa-40d7-ab02-e25379a096c0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.082890 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.082983 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.083038 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.083735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.083716 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.085298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.085277 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.085702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.085676 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8367b1-00aa-40d7-ab02-e25379a096c0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.085802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.085716 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.086024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.086003 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.107335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.107301 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8367b1-00aa-40d7-ab02-e25379a096c0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.107565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.107545 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4mc\" (UniqueName: \"kubernetes.io/projected/3c8367b1-00aa-40d7-ab02-e25379a096c0-kube-api-access-5n4mc\") pod \"istiod-openshift-gateway-7ddd9cfdc8-vkwdx\" (UID: \"3c8367b1-00aa-40d7-ab02-e25379a096c0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.272007 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.271979 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:27.608964 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:04:27.608934 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8367b1_00aa_40d7_ab02_e25379a096c0.slice/crio-c57299a40d92765143603cabdf924b147f23766fe60be3dd8dad2ef71a05d3f1 WatchSource:0}: Error finding container c57299a40d92765143603cabdf924b147f23766fe60be3dd8dad2ef71a05d3f1: Status 404 returned error can't find the container with id c57299a40d92765143603cabdf924b147f23766fe60be3dd8dad2ef71a05d3f1 Oct 30 15:04:27.609086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.609028 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx"] Oct 30 15:04:27.611078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.611062 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:04:27.880764 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:27.880702 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" event={"ID":"3c8367b1-00aa-40d7-ab02-e25379a096c0","Type":"ContainerStarted","Data":"c57299a40d92765143603cabdf924b147f23766fe60be3dd8dad2ef71a05d3f1"} Oct 30 15:04:30.545770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:30.545728 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:04:30.546073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:30.545813 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:04:30.907755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:30.907718 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" event={"ID":"3c8367b1-00aa-40d7-ab02-e25379a096c0","Type":"ContainerStarted","Data":"81b978d75c41e7c72995afc1921eebbc1aae66258ca720cfb9f03ced42315600"} Oct 30 15:04:30.907967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:30.907951 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:30.971892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:30.971843 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" podStartSLOduration=2.038628752 podStartE2EDuration="4.97183068s" totalImagesPullingTime="2.933201928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:04:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:04:30.970029462 +0000 UTC m=+974.929070867" watchObservedRunningTime="2026-10-30 15:04:30.97183068 +0000 UTC m=+974.930872086" Oct 30 15:04:31.916939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:31.916911 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-vkwdx" Oct 30 15:04:34.166375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:34.165458 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:34.348374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:34.348195 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:04:36.390743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.390710 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq"] Oct 30 15:04:36.394954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.394933 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.419089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.419065 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq"] Oct 30 15:04:36.428593 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.428566 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:04:36.428714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.428658 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:04:36.450918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.450894 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.451066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.451048 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.451146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.451080 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zvh\" (UniqueName: \"kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.552491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.552462 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.552618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.552506 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zvh\" (UniqueName: \"kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.552692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.552670 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.553297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.553279 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.554847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.554826 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.580408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.580382 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zvh\" (UniqueName: \"kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh\") pod \"maas-api-revoke-sub-4534535b-kn2kq\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:36.708890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:36.708829 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:37.011765 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:04:37.011735 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0e3e66_c0bf_42e5_a786_3f78e5bb7869.slice/crio-da396e1afb42d301d1ea759aa2267ac2d1fa9284bf82529b60be437c1fb09d8b WatchSource:0}: Error finding container da396e1afb42d301d1ea759aa2267ac2d1fa9284bf82529b60be437c1fb09d8b: Status 404 returned error can't find the container with id da396e1afb42d301d1ea759aa2267ac2d1fa9284bf82529b60be437c1fb09d8b Oct 30 15:04:37.012048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:37.012026 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq"] Oct 30 15:04:37.991072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:37.991042 6554 generic.go:356] "Generic (PLEG): container finished" podID="5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" containerID="a4f2f934f5eb83b5f0be1c9f2f6877b0728c90c62655af3f0e84d95936a2f7d1" exitCode=0 Oct 30 15:04:37.991467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:37.991103 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" event={"ID":"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869","Type":"ContainerDied","Data":"a4f2f934f5eb83b5f0be1c9f2f6877b0728c90c62655af3f0e84d95936a2f7d1"} Oct 30 15:04:37.991467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:37.991215 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" event={"ID":"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869","Type":"ContainerStarted","Data":"da396e1afb42d301d1ea759aa2267ac2d1fa9284bf82529b60be437c1fb09d8b"} Oct 30 15:04:39.142232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.142211 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:39.182963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.182935 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh\" (UniqueName: \"kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh\") pod \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " Oct 30 15:04:39.183103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.183072 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca\") pod \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " Oct 30 15:04:39.183178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.183160 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca\") pod \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\" (UID: \"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869\") " Oct 30 15:04:39.183481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.183450 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca" pod "5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" (UID: "5b0e3e66-c0bf-42e5-a786-3f78e5bb7869"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:04:39.183816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.183794 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:39.185140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.185106 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh" pod "5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" (UID: "5b0e3e66-c0bf-42e5-a786-3f78e5bb7869"). InnerVolumeSpecName "kube-api-access-x2zvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:04:39.185219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.185200 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca" pod "5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" (UID: "5b0e3e66-c0bf-42e5-a786-3f78e5bb7869"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:04:39.205166 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:04:39.205114 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:04:39.285503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.285443 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:39.285503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:39.285467 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zvh\" (UniqueName: \"kubernetes.io/projected/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869-kube-api-access-x2zvh\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:04:40.005485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:40.005454 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" Oct 30 15:04:40.005662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:40.005489 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq" event={"ID":"5b0e3e66-c0bf-42e5-a786-3f78e5bb7869","Type":"ContainerDied","Data":"da396e1afb42d301d1ea759aa2267ac2d1fa9284bf82529b60be437c1fb09d8b"} Oct 30 15:04:40.005662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:40.005537 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da396e1afb42d301d1ea759aa2267ac2d1fa9284bf82529b60be437c1fb09d8b" Oct 30 15:04:56.354375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.351196 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.404375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.402431 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.455627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.455585 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.581477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.581433 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.614940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.585435 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:56.673724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.585468 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:04:56.674392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:56.586160 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.085482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.085434 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.121940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.121899 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.178493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.178452 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:04:57.351542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.351437 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.352550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.352527 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:04:57.407374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.405457 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.407374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.406550 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:04:57.474444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.458926 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:04:57.474812 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.474786 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:04:57.664447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.614897 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.664447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.616317 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:04:57.743551 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.743501 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:04:57.786374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.745668 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:04:57.963424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.776453 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:57.963424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.777506 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:04:57.963424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.952913 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:04:57.963424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:57.952956 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:04:58.282471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:58.267417 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:58.283332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:58.283304 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:04:58.476244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:58.452703 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:04:58.476244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:58.462512 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:04:58.476244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:58.464194 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:04:59.006379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:58.996153 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:04:59.031263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.011609 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.086119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.086078 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.242375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.235219 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:04:59.264526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.264429 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.267309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.267275 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.268599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.268572 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.269492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.268684 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.290896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.290868 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:04:59.503529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.503467 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:04:59.515447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.513522 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:04:59.515447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.513887 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:04:59.746512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:04:59.746468 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.117374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.105416 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.142712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.117668 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:00.274330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.273449 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:05:00.274330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.273800 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 15:05:00.274330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.274076 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 15:05:00.290266 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.290210 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.525597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.525543 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:05:00.560386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.530243 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:00.560386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.530968 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:05:00.682272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.682240 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:00.717765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:00.717725 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.137890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.137860 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:01.142444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.142411 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.143145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.143116 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:01.154362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.143807 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:01.154362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.147565 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:01.167587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.167565 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:01.178096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.178074 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:01.206888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.206867 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:01.223525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.223495 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:01.240898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:01.240872 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:05.207054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.207018 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s"] Oct 30 15:05:05.207646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.207630 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" containerName="revoke-keys" Oct 30 15:05:05.207646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.207645 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" containerName="revoke-keys" Oct 30 15:05:05.207763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.207732 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" containerName="revoke-keys" Oct 30 15:05:05.211268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.211246 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.230103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.230073 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s"] Oct 30 15:05:05.240189 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.240161 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:05:05.240328 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.240225 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:05:05.295961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.295933 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.296084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.295995 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.296124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.296091 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmcc\" (UniqueName: \"kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.397413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.397390 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.397536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.397446 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.397536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.397507 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmcc\" (UniqueName: \"kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.398144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.398122 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.399697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.399679 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.422549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.422525 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmcc\" (UniqueName: \"kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc\") pod \"maas-api-revoke-sub-6953b4e4-xvm6s\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.524719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.524665 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:05.841545 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:05:05.832814 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd890d9c1_7385_474a_bcec_244f7ec89ea8.slice/crio-9735f13c4db2dcea03978208a68cc213c76f1795820cc331762709f1ef6be21a WatchSource:0}: Error finding container 9735f13c4db2dcea03978208a68cc213c76f1795820cc331762709f1ef6be21a: Status 404 returned error can't find the container with id 9735f13c4db2dcea03978208a68cc213c76f1795820cc331762709f1ef6be21a Oct 30 15:05:05.841545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:05.832921 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s"] Oct 30 15:05:06.311609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:06.311572 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" event={"ID":"d890d9c1-7385-474a-bcec-244f7ec89ea8","Type":"ContainerStarted","Data":"e7bf4b82cfb77148a62795d05badc1fac0dadaf10a013c723e56db55ddbdd4cf"} Oct 30 15:05:06.312091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:06.311632 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" event={"ID":"d890d9c1-7385-474a-bcec-244f7ec89ea8","Type":"ContainerStarted","Data":"9735f13c4db2dcea03978208a68cc213c76f1795820cc331762709f1ef6be21a"} Oct 30 15:05:06.355605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:06.355542 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" podStartSLOduration=1.35552029 podStartE2EDuration="1.35552029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:06.354808469 +0000 UTC m=+1010.313849877" watchObservedRunningTime="2026-10-30 15:05:06.35552029 +0000 UTC m=+1010.314561699" Oct 30 15:05:07.319857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:07.319822 6554 generic.go:356] "Generic (PLEG): container finished" podID="d890d9c1-7385-474a-bcec-244f7ec89ea8" containerID="e7bf4b82cfb77148a62795d05badc1fac0dadaf10a013c723e56db55ddbdd4cf" exitCode=0 Oct 30 15:05:07.320262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:07.319909 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" event={"ID":"d890d9c1-7385-474a-bcec-244f7ec89ea8","Type":"ContainerDied","Data":"e7bf4b82cfb77148a62795d05badc1fac0dadaf10a013c723e56db55ddbdd4cf"} Oct 30 15:05:08.496164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.496139 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:08.538842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.538819 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca\") pod \"d890d9c1-7385-474a-bcec-244f7ec89ea8\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " Oct 30 15:05:08.538984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.538899 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca\") pod \"d890d9c1-7385-474a-bcec-244f7ec89ea8\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " Oct 30 15:05:08.539050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.538995 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc\" (UniqueName: \"kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc\") pod \"d890d9c1-7385-474a-bcec-244f7ec89ea8\" (UID: \"d890d9c1-7385-474a-bcec-244f7ec89ea8\") " Oct 30 15:05:08.539228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.539204 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca" pod "d890d9c1-7385-474a-bcec-244f7ec89ea8" (UID: "d890d9c1-7385-474a-bcec-244f7ec89ea8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:08.540983 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.540948 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:05:08.558325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.558290 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca" pod "d890d9c1-7385-474a-bcec-244f7ec89ea8" (UID: "d890d9c1-7385-474a-bcec-244f7ec89ea8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:08.558472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.558421 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc" pod "d890d9c1-7385-474a-bcec-244f7ec89ea8" (UID: "d890d9c1-7385-474a-bcec-244f7ec89ea8"). InnerVolumeSpecName "kube-api-access-xmmcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:08.642542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.642523 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d890d9c1-7385-474a-bcec-244f7ec89ea8-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:05:08.642638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:08.642544 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmcc\" (UniqueName: \"kubernetes.io/projected/d890d9c1-7385-474a-bcec-244f7ec89ea8-kube-api-access-xmmcc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:05:09.201133 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:05:09.201103 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:05:09.335910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:09.335882 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" Oct 30 15:05:09.336058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:09.335910 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s" event={"ID":"d890d9c1-7385-474a-bcec-244f7ec89ea8","Type":"ContainerDied","Data":"9735f13c4db2dcea03978208a68cc213c76f1795820cc331762709f1ef6be21a"} Oct 30 15:05:09.336058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:09.335970 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9735f13c4db2dcea03978208a68cc213c76f1795820cc331762709f1ef6be21a" Oct 30 15:05:18.036493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.035420 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.141374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.140446 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.145373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.144465 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 15:05:18.267486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.267439 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.294428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.269253 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.295591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.280151 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.296777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.280850 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.366374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.365852 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:18.366374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.365932 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.536500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.536435 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.537857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.536773 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:18.867384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:18.865383 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 15:05:19.023509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.023460 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 15:05:19.049703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.049655 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.152432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.145432 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.152432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.146508 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:19.152432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.146568 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:19.152432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.145471 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:05:19.152432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.148660 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:19.272373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.268542 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.345712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.345667 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:05:19.345906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.345765 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.345906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.345813 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.412422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.407455 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:19.412422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.407533 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:19.412422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.407572 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:19.412422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.407608 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:19.605402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.602945 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.606165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.606138 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:19.821425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.798469 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded" Oct 30 15:05:19.823093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.822615 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:19.823093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.822720 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:19.823343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.823324 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:19.823467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:19.823450 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.021331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.021260 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.028917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.028889 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:20.108427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.061118 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:20.108427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.061676 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.193704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.166992 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.223574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.223525 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:20.249500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.249459 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 15:05:20.395434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.368624 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.397874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.397835 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:05:20.470948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.444643 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 15:05:20.470948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.449544 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 15:05:20.764602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.764487 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:20.963405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.942597 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:05:21.041426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:20.988103 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:21.041426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.028151 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.058378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.054718 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.261236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.260671 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:21.279949 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:05:21.270795 6554 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.112s" Oct 30 15:05:21.279949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.270926 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:21.279949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.270971 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:21.279949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.271000 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:21.279949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.271047 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:21.283643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.282477 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:21.285975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.285952 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:21.326011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.325937 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:21.337311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.337146 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:21.362406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:21.360441 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:27.074914 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.074880 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk"] Oct 30 15:05:27.075563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.075548 6554 cpu_manager.go:414] "containerMap: removing container" podUID="d890d9c1-7385-474a-bcec-244f7ec89ea8" containerName="revoke-keys" Oct 30 15:05:27.075626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.075564 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d890d9c1-7385-474a-bcec-244f7ec89ea8" containerName="revoke-keys" Oct 30 15:05:27.075661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.075654 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="d890d9c1-7385-474a-bcec-244f7ec89ea8" containerName="revoke-keys" Oct 30 15:05:27.080324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.080304 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.111429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.111398 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.111618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.111599 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.111709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.111663 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fj5b\" (UniqueName: \"kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.112718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.112695 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:05:27.112817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.112716 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:05:27.116560 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.116540 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk"] Oct 30 15:05:27.213404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.213379 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.213540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.213414 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.213540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.213457 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fj5b\" (UniqueName: \"kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.214039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.214019 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.215820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.215799 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.238024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.238004 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fj5b\" (UniqueName: \"kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b\") pod \"maas-api-revoke-sub-db4e1d9c-hsvzk\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.393893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.393873 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:27.701763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:27.701729 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk"] Oct 30 15:05:27.708025 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:05:27.708000 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c53d18e_a065_4428_a006_e7b90d104548.slice/crio-804e2e3c6b9666107ee0bf499d2a5855e4dfb3ee8121d37bec1c5d2ca7ac4764 WatchSource:0}: Error finding container 804e2e3c6b9666107ee0bf499d2a5855e4dfb3ee8121d37bec1c5d2ca7ac4764: Status 404 returned error can't find the container with id 804e2e3c6b9666107ee0bf499d2a5855e4dfb3ee8121d37bec1c5d2ca7ac4764 Oct 30 15:05:28.112584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:28.112552 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" event={"ID":"7c53d18e-a065-4428-a006-e7b90d104548","Type":"ContainerStarted","Data":"d3ea8a15d1466995e50b5e5c760979502d2a10d7774e6981d7e7110476e7835a"} Oct 30 15:05:28.112911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:28.112602 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" event={"ID":"7c53d18e-a065-4428-a006-e7b90d104548","Type":"ContainerStarted","Data":"804e2e3c6b9666107ee0bf499d2a5855e4dfb3ee8121d37bec1c5d2ca7ac4764"} Oct 30 15:05:28.157162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:28.157118 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" podStartSLOduration=1.157104977 podStartE2EDuration="1.157104977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:28.156025869 +0000 UTC m=+1032.115067283" watchObservedRunningTime="2026-10-30 15:05:28.157104977 +0000 UTC m=+1032.116146381" Oct 30 15:05:29.121047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:29.121015 6554 generic.go:356] "Generic (PLEG): container finished" podID="7c53d18e-a065-4428-a006-e7b90d104548" containerID="d3ea8a15d1466995e50b5e5c760979502d2a10d7774e6981d7e7110476e7835a" exitCode=0 Oct 30 15:05:29.121513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:29.121083 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" event={"ID":"7c53d18e-a065-4428-a006-e7b90d104548","Type":"ContainerDied","Data":"d3ea8a15d1466995e50b5e5c760979502d2a10d7774e6981d7e7110476e7835a"} Oct 30 15:05:30.300010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.299986 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:30.353839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.353810 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca\") pod \"7c53d18e-a065-4428-a006-e7b90d104548\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " Oct 30 15:05:30.354179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.354157 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca\") pod \"7c53d18e-a065-4428-a006-e7b90d104548\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " Oct 30 15:05:30.354476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.354455 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b\" (UniqueName: \"kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b\") pod \"7c53d18e-a065-4428-a006-e7b90d104548\" (UID: \"7c53d18e-a065-4428-a006-e7b90d104548\") " Oct 30 15:05:30.358447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.358389 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca" pod "7c53d18e-a065-4428-a006-e7b90d104548" (UID: "7c53d18e-a065-4428-a006-e7b90d104548"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:05:30.360115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.360072 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca" pod "7c53d18e-a065-4428-a006-e7b90d104548" (UID: "7c53d18e-a065-4428-a006-e7b90d104548"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:05:30.369426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.369396 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b" pod "7c53d18e-a065-4428-a006-e7b90d104548" (UID: "7c53d18e-a065-4428-a006-e7b90d104548"). InnerVolumeSpecName "kube-api-access-6fj5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:05:30.456723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.456683 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fj5b\" (UniqueName: \"kubernetes.io/projected/7c53d18e-a065-4428-a006-e7b90d104548-kube-api-access-6fj5b\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:05:30.456723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.456724 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:05:30.456723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:30.456739 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c53d18e-a065-4428-a006-e7b90d104548-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:05:31.137803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:31.137777 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" Oct 30 15:05:31.137965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:31.137770 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk" event={"ID":"7c53d18e-a065-4428-a006-e7b90d104548","Type":"ContainerDied","Data":"804e2e3c6b9666107ee0bf499d2a5855e4dfb3ee8121d37bec1c5d2ca7ac4764"} Oct 30 15:05:31.137965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:31.137919 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="804e2e3c6b9666107ee0bf499d2a5855e4dfb3ee8121d37bec1c5d2ca7ac4764" Oct 30 15:05:33.101370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.097138 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.101370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.098518 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.199285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.199244 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:05:33.385232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.357920 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.385232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.380251 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.534376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.510618 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:33.718376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.718194 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:33.718376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:33.718264 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.097821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.097718 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:34.098534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.098510 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:34.099448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.098706 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:05:34.100500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.099800 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:34.366439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.365902 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.411376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.400818 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 15:05:34.426372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.418910 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.435505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.418423 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.436241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.436218 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:34.436835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.436818 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:34.564373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.519842 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 15:05:34.835864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.835825 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:34.836158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.836125 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:34.998377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.984444 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:34.998377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:34.984508 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.102428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.102193 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.104380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.103843 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:35.277402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.265431 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.367460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.366420 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.367460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.366633 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:35.417024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.416919 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.418189 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.417299 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:35.426511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.426480 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.439889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.439859 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.692429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.588102 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:35.692429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.591811 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded" Oct 30 15:05:35.692429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:35.675206 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:36.109719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.109613 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.110364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.110305 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.288373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.288230 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:36.288373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.288313 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.334637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.334600 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.375678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.375590 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.430869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.430817 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:05:36.439378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.431107 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.464394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.441670 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.464664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.447858 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 15:05:36.464847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.447887 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:05:36.626420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.626314 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:36.632880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.632854 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:36.692269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.691495 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:36.799687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.799647 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:36.799989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.799970 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:36.800196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.800173 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:36.800373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.800337 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:36.800543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.800530 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:36.800697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.800684 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:36.832382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.831883 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:36.853330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.853304 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:36.881637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:36.881553 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:39.219745 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:05:39.219695 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:05:46.452969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.452938 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb"] Oct 30 15:05:46.453622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.453605 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7c53d18e-a065-4428-a006-e7b90d104548" containerName="revoke-keys" Oct 30 15:05:46.453622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.453623 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c53d18e-a065-4428-a006-e7b90d104548" containerName="revoke-keys" Oct 30 15:05:46.453766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.453756 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c53d18e-a065-4428-a006-e7b90d104548" containerName="revoke-keys" Oct 30 15:05:46.457407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.457390 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.484410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.484344 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb"] Oct 30 15:05:46.487917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.487896 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dc34df04-dockercfg-6nmmd\"" Oct 30 15:05:46.488050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.487972 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dc34df04\"" Oct 30 15:05:46.540679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.540643 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-255mt\" (UniqueName: \"kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt\") pod \"payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.540862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.540775 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.551645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.551602 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82"] Oct 30 15:05:46.557224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.557157 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.588176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.585376 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82"] Oct 30 15:05:46.673798 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.673758 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kksd\" (UniqueName: \"kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd\") pod \"payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.674217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.674190 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-255mt\" (UniqueName: \"kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt\") pod \"payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.674435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.674417 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.674629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.674611 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.676634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.676154 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.708597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.708522 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-255mt\" (UniqueName: \"kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt\") pod \"payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.773783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.773754 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:46.775939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.775911 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.776122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.776049 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kksd\" (UniqueName: \"kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd\") pod \"payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.777120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.776858 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume\") pod \"payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.801691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.801662 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kksd\" (UniqueName: \"kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd\") pod \"payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:46.882680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:46.882655 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:47.085399 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:05:47.085339 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eceea3c_3645_4ea9_86a9_3215761db08e.slice/crio-a824c9d0b9449a57cbd0a615ea8c118e6132fb6ed1ed67d4c15b3e83dccc5bc8 WatchSource:0}: Error finding container a824c9d0b9449a57cbd0a615ea8c118e6132fb6ed1ed67d4c15b3e83dccc5bc8: Status 404 returned error can't find the container with id a824c9d0b9449a57cbd0a615ea8c118e6132fb6ed1ed67d4c15b3e83dccc5bc8 Oct 30 15:05:47.085650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.085628 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb"] Oct 30 15:05:47.200294 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:05:47.200266 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ffa6e8_803e_46c3_bc5a_ff5ba55e6aca.slice/crio-9500d62138ba2e35888a42fdea3bb8bee98a6c25347194b3ae041302df0bef1c WatchSource:0}: Error finding container 9500d62138ba2e35888a42fdea3bb8bee98a6c25347194b3ae041302df0bef1c: Status 404 returned error can't find the container with id 9500d62138ba2e35888a42fdea3bb8bee98a6c25347194b3ae041302df0bef1c Oct 30 15:05:47.201772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.201744 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82"] Oct 30 15:05:47.710333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.710295 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" event={"ID":"0eceea3c-3645-4ea9-86a9-3215761db08e","Type":"ContainerStarted","Data":"492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac"} Oct 30 15:05:47.710805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.710374 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" event={"ID":"0eceea3c-3645-4ea9-86a9-3215761db08e","Type":"ContainerStarted","Data":"a824c9d0b9449a57cbd0a615ea8c118e6132fb6ed1ed67d4c15b3e83dccc5bc8"} Oct 30 15:05:47.710805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.710407 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:47.712207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.712182 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" event={"ID":"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca","Type":"ContainerStarted","Data":"ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae"} Oct 30 15:05:47.712363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.712228 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" event={"ID":"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca","Type":"ContainerStarted","Data":"9500d62138ba2e35888a42fdea3bb8bee98a6c25347194b3ae041302df0bef1c"} Oct 30 15:05:47.712363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.712255 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:47.754915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.754870 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" podStartSLOduration=1.7548576329999999 podStartE2EDuration="1.754857633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:47.753611271 +0000 UTC m=+1051.712652667" watchObservedRunningTime="2026-10-30 15:05:47.754857633 +0000 UTC m=+1051.713899069" Oct 30 15:05:47.799662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:47.799603 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" podStartSLOduration=1.799585655 podStartE2EDuration="1.799585655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:47.796483824 +0000 UTC m=+1051.755525229" watchObservedRunningTime="2026-10-30 15:05:47.799585655 +0000 UTC m=+1051.758627060" Oct 30 15:05:52.476478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:52.476429 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:05:52.786458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:52.783708 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:52.799921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:52.799881 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:52.858656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:52.856618 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.106428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.105557 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.106428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.106026 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:05:53.203376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.197427 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.474477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.474429 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.475148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.474742 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.489463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.489425 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.491600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.491573 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:05:53.492173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.492154 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:53.788429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.784481 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 15:05:53.833815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.794467 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:53.833815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.823725 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:53.833815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.824318 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:53.906385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:53.898696 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 30 15:05:54.198513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.169231 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:54.198513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.169663 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:05:54.206375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.202010 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.417376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.410280 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:54.417376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.410399 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:54.417376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.410464 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:54.487423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.486624 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.511385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.504593 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:05:54.511385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.504672 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:05:54.580374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.556982 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:54.580374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.557052 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:54.799433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.795523 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.835380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.834595 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:54.850501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:54.850461 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.214376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.214089 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.239372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.214767 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:55.253375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.247064 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.552428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.551473 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.552428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.551854 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 15:05:55.552428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.552132 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:05:55.798409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.795711 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.842452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.840418 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.854440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.851424 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:55.854440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.852079 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:55.976799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:55.976764 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf"] Oct 30 15:05:56.097043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.096966 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf"] Oct 30 15:05:56.102368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.097328 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.258469 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.258908 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.258931 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.265785 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwhl\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.268979 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269039 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269109 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269150 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269203 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269290 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.269390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269331 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.270090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.269434 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.366162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.366089 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:05:56.391498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.378305 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.391787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.391766 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.401107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.401082 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.401340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.401320 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.436395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.401988 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.437930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.405773 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.455367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.438172 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.456299 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.456272 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.456606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.456584 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.456835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.456815 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwhl\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.457008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.456982 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.462366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.457625 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.505420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.462792 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.505618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.463470 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.582371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.561531 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.586368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.584680 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:05:56.657906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.644233 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": context deadline exceeded" Oct 30 15:05:56.657906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.644563 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 15:05:56.676372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.666945 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:05:56.699929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.699898 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.712975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.712148 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwhl\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.712975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.712302 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:05:56.758289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.750668 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:05:56.760956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.760930 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token\") pod \"e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:56.782536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.782501 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:05:56.807423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.807400 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:05:56.846051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.846021 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:05:56.848791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.848771 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" Oct 30 15:05:56.965978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:56.962815 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:57.312255 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:05:57.312227 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b9e7c8_9e0d_4a10_a3c5_3906077e22aa.slice/crio-28178aa86194fc1d0991f863d79f2bdd1c92dd901ee9ba1377e591c3ba89649a WatchSource:0}: Error finding container 28178aa86194fc1d0991f863d79f2bdd1c92dd901ee9ba1377e591c3ba89649a: Status 404 returned error can't find the container with id 28178aa86194fc1d0991f863d79f2bdd1c92dd901ee9ba1377e591c3ba89649a Oct 30 15:05:57.312498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:57.312476 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf"] Oct 30 15:05:57.314941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:57.314905 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:05:57.315064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:57.314984 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:05:57.315064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:57.315030 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:05:58.307191 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.307152 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" event={"ID":"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa","Type":"ContainerStarted","Data":"a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e"} Oct 30 15:05:58.307606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.307215 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" event={"ID":"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa","Type":"ContainerStarted","Data":"28178aa86194fc1d0991f863d79f2bdd1c92dd901ee9ba1377e591c3ba89649a"} Oct 30 15:05:58.382230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.382161 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podStartSLOduration=4.382143161 podStartE2EDuration="4.382143161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:05:58.380369521 +0000 UTC m=+1062.339410927" watchObservedRunningTime="2026-10-30 15:05:58.382143161 +0000 UTC m=+1062.341184570" Oct 30 15:05:58.722019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.721990 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:05:58.722181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.722059 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:05:58.977481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.976876 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:05:58.977481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.977045 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:05:58.977481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:58.977089 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 15:05:59.969006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:59.967009 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:05:59.969006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:05:59.967068 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 15:06:00.976373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:00.975445 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:06:00.976373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:00.975516 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 15:06:01.970713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:01.970653 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:06:01.971000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:01.970973 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 15:06:02.971375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:02.967738 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:06:02.971375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:02.967794 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 15:06:03.966945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:03.966903 6554 patch_prober.go:29] interesting pod/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 15:06:03.967162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:03.966974 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 15:06:04.971377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:04.971326 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:06:04.971795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:04.971773 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:06:04.972544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:04.972526 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:06:06.372898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.372869 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94"] Oct 30 15:06:06.377148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.377128 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.393407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.393386 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dc34df04-serving-cert\"" Oct 30 15:06:06.411764 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.411737 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94"] Oct 30 15:06:06.468468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.468436 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqd7\" (UniqueName: \"kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.468599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.468557 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.468845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.468830 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.468962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.468945 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.572020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.571993 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.572382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.572330 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqd7\" (UniqueName: \"kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.572611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.572587 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.572758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.572661 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.573007 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.572983 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.573507 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:06.572849 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-serving-cert: secret "maas-api-e2e-ait-dc34df04-serving-cert" not found Oct 30 15:06:06.575372 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:06.575337 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls podName:1a01e9b9-86e6-4ef3-aed6-ff07530cf464 nodeName:}" failed. No retries permitted until 2026-10-30 15:06:07.075316672 +0000 UTC m=+1071.034358074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls") pod "maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" (UID: "1a01e9b9-86e6-4ef3-aed6-ff07530cf464") : secret "maas-api-e2e-ait-dc34df04-serving-cert" not found Oct 30 15:06:06.577055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.577030 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:06.599581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:06.598935 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqd7\" (UniqueName: \"kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:07.080264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:07.080230 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:07.082744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:07.082724 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\") pod \"maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:07.291795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:07.291767 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:07.615837 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:06:07.615807 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a01e9b9_86e6_4ef3_aed6_ff07530cf464.slice/crio-a9fcb71832aafaefd0b4f226e6fd7ef5e9fe2d90e8c78001a25a3242a4a5c15b WatchSource:0}: Error finding container a9fcb71832aafaefd0b4f226e6fd7ef5e9fe2d90e8c78001a25a3242a4a5c15b: Status 404 returned error can't find the container with id a9fcb71832aafaefd0b4f226e6fd7ef5e9fe2d90e8c78001a25a3242a4a5c15b Oct 30 15:06:07.616239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:07.616022 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94"] Oct 30 15:06:08.405253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:08.405216 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" event={"ID":"1a01e9b9-86e6-4ef3-aed6-ff07530cf464","Type":"ContainerStarted","Data":"59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131"} Oct 30 15:06:08.405476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:08.405276 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" event={"ID":"1a01e9b9-86e6-4ef3-aed6-ff07530cf464","Type":"ContainerStarted","Data":"a9fcb71832aafaefd0b4f226e6fd7ef5e9fe2d90e8c78001a25a3242a4a5c15b"} Oct 30 15:06:08.405476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:08.405304 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:08.450416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:08.450367 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" podStartSLOduration=2.133118655 podStartE2EDuration="2.450338418s" totalImagesPullingTime="317.219763ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:08.449578236 +0000 UTC m=+1072.408619641" watchObservedRunningTime="2026-10-30 15:06:08.450338418 +0000 UTC m=+1072.409379822" Oct 30 15:06:09.202027 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:09.201992 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:06:14.417688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:14.417660 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:26.033033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:26.032992 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h"] Oct 30 15:06:26.046070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:26.046040 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c06022e8-27k4h"] Oct 30 15:06:26.156316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:26.156285 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad12281b-dbb6-451d-8190-d6bd576cbf25" path="/var/lib/kubelet/pods/ad12281b-dbb6-451d-8190-d6bd576cbf25/volumes" Oct 30 15:06:34.034642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:34.034568 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm"] Oct 30 15:06:34.044947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:34.044920 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81e8d13-8ldsm"] Oct 30 15:06:34.156621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:34.156591 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="394db93e-3e6f-42a5-b356-a09ca41483fe" path="/var/lib/kubelet/pods/394db93e-3e6f-42a5-b356-a09ca41483fe/volumes" Oct 30 15:06:37.013405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.013374 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm"] Oct 30 15:06:37.017890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.017868 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.032387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.032366 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn"] Oct 30 15:06:37.036554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.036533 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.042509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.042465 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khr2k\"" Oct 30 15:06:37.047308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.047285 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm"] Oct 30 15:06:37.061765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.061745 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn"] Oct 30 15:06:37.173679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173651 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sf2z\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.173820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173685 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.173820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173712 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.173905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173816 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.173905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173849 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.173978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173925 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw2qs\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.173978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.173948 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174018 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.174098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174061 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.174171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174134 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174179 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174204 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174279 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.174335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174310 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174328 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.174449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174369 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174434 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.174524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.174468 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.275939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.275872 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.275939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.275907 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.275939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.275931 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.275955 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.275975 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276003 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276058 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276099 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276152 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sf2z\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276195 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276237 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276280 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276306 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276337 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276339 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276424 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276434 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw2qs\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276501 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276518 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.276576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276545 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.277108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276605 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.277108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276619 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.277108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.276691 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.277108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.277069 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.277327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.277226 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.277412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.277327 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.277486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.277440 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.277542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.277490 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.279256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.279215 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.279342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.279288 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.279472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.279449 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.279785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.279765 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.301645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.301620 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw2qs\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.301776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.301668 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sf2z\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.301776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.301740 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2jsbm\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.301776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.301757 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-2cfhn\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.333478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.333458 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:37.352097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.352078 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:37.678434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.677402 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn"] Oct 30 15:06:37.679632 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:06:37.679588 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9e976d_ce6e_48a4_a171_a46075af6ed1.slice/crio-2fed1783dac8a35344fbc7330b3426c5ea9cb9372458b87d65fd02e40185abf9 WatchSource:0}: Error finding container 2fed1783dac8a35344fbc7330b3426c5ea9cb9372458b87d65fd02e40185abf9: Status 404 returned error can't find the container with id 2fed1783dac8a35344fbc7330b3426c5ea9cb9372458b87d65fd02e40185abf9 Oct 30 15:06:37.682051 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:06:37.682023 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7b09e4_9eda_460a_816a_5686eebcbbd5.slice/crio-d8ebff62621b54cdad14f48410d2ce1ddd46a7e1cfb59ee1080ad870fa566f87 WatchSource:0}: Error finding container d8ebff62621b54cdad14f48410d2ce1ddd46a7e1cfb59ee1080ad870fa566f87: Status 404 returned error can't find the container with id d8ebff62621b54cdad14f48410d2ce1ddd46a7e1cfb59ee1080ad870fa566f87 Oct 30 15:06:37.683418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.683369 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm"] Oct 30 15:06:37.684272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.684238 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:06:37.684407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.684302 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:06:37.684407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.684333 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:06:37.685514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.685485 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:06:37.685613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.685544 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:06:37.685613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:37.685573 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:06:38.624976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:38.624941 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" event={"ID":"ec9e976d-ce6e-48a4-a171-a46075af6ed1","Type":"ContainerStarted","Data":"ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323"} Oct 30 15:06:38.625443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:38.625012 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" event={"ID":"ec9e976d-ce6e-48a4-a171-a46075af6ed1","Type":"ContainerStarted","Data":"2fed1783dac8a35344fbc7330b3426c5ea9cb9372458b87d65fd02e40185abf9"} Oct 30 15:06:38.626622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:38.626593 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" event={"ID":"7e7b09e4-9eda-460a-816a-5686eebcbbd5","Type":"ContainerStarted","Data":"c0d4f7a0175551227072147df4da726e1ed8bf0cbe9f729905fd1a98a8563e88"} Oct 30 15:06:38.626749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:38.626636 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" event={"ID":"7e7b09e4-9eda-460a-816a-5686eebcbbd5","Type":"ContainerStarted","Data":"d8ebff62621b54cdad14f48410d2ce1ddd46a7e1cfb59ee1080ad870fa566f87"} Oct 30 15:06:38.722320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:38.722271 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podStartSLOduration=2.7222574010000002 podStartE2EDuration="2.722257401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:38.717608826 +0000 UTC m=+1102.676650233" watchObservedRunningTime="2026-10-30 15:06:38.722257401 +0000 UTC m=+1102.681298805" Oct 30 15:06:38.813313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:38.813257 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podStartSLOduration=1.813241873 podStartE2EDuration="1.813241873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:06:38.772965403 +0000 UTC m=+1102.732006808" watchObservedRunningTime="2026-10-30 15:06:38.813241873 +0000 UTC m=+1102.772283277" Oct 30 15:06:39.230343 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:39.230291 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:06:39.333744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:39.333714 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:39.335225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:39.335198 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:39.335400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:39.335252 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:39.353274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:39.353245 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:39.366523 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:39.366474 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:39.366807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:39.366778 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:40.333794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:40.333749 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:40.334208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:40.333808 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:40.353100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:40.353076 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:40.353251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:40.353233 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:41.334426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:41.334392 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:41.334834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:41.334448 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:41.361437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:41.353340 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:41.361671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:41.361645 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:42.334498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:42.334464 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:42.334944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:42.334517 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:42.353024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:42.352984 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:42.353139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:42.353067 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:43.334104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:43.334066 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:43.334289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:43.334123 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:43.352378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:43.352331 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:43.352813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:43.352785 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:44.333867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:44.333827 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:44.334078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:44.333893 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:44.352703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:44.352675 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:44.353131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:44.352726 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:45.334247 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:45.334206 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 15:06:45.334449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:45.334274 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 15:06:45.352650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:45.352622 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body= Oct 30 15:06:45.352769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:45.352687 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" Oct 30 15:06:46.488449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:46.488414 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82"] Oct 30 15:06:46.538604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:46.538569 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb"] Oct 30 15:06:47.386158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.386117 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2jsbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:06:47.386533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.386498 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:06:47.398335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.398294 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:06:47.398645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.398611 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:06:47.717593 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.717495 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:47.719933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.719703 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:47.726592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.726570 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:06:47.792254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.792224 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:47.803370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.794437 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:47.804684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.803576 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:06:47.817382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.816920 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:06:47.842780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:47.841151 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:06:51.054529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.054491 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg"] Oct 30 15:06:51.066316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.066282 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee54494-nbmxg"] Oct 30 15:06:51.489626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.489549 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" podUID="76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" containerName="payload-processing" containerID="cri-o://ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae" gracePeriod=25 Oct 30 15:06:51.539644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.539592 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" podUID="0eceea3c-3645-4ea9-86a9-3215761db08e" containerName="payload-pre-processing" containerID="cri-o://492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac" gracePeriod=25 Oct 30 15:06:51.838070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.838038 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:06:51.858404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.858385 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:06:51.896847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.896821 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd\" (UniqueName: \"kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd\") pod \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " Oct 30 15:06:51.897031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.897013 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume\") pod \"0eceea3c-3645-4ea9-86a9-3215761db08e\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " Oct 30 15:06:51.897128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.897107 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume\") pod \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\" (UID: \"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca\") " Oct 30 15:06:51.897316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.897296 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt\" (UniqueName: \"kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt\") pod \"0eceea3c-3645-4ea9-86a9-3215761db08e\" (UID: \"0eceea3c-3645-4ea9-86a9-3215761db08e\") " Oct 30 15:06:51.897694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.897422 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume" pod "0eceea3c-3645-4ea9-86a9-3215761db08e" (UID: "0eceea3c-3645-4ea9-86a9-3215761db08e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:51.897694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.897485 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume" pod "76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" (UID: "76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:51.898072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.898046 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eceea3c-3645-4ea9-86a9-3215761db08e-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:51.898143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.898076 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:51.899260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.899234 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd" pod "76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" (UID: "76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca"). InnerVolumeSpecName "kube-api-access-7kksd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:51.899618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.899598 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt" pod "0eceea3c-3645-4ea9-86a9-3215761db08e" (UID: "0eceea3c-3645-4ea9-86a9-3215761db08e"). InnerVolumeSpecName "kube-api-access-255mt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:51.999266 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.999234 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-255mt\" (UniqueName: \"kubernetes.io/projected/0eceea3c-3645-4ea9-86a9-3215761db08e-kube-api-access-255mt\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:51.999266 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:51.999263 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kksd\" (UniqueName: \"kubernetes.io/projected/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca-kube-api-access-7kksd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:52.155555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.155523 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10a2fbdf-9d64-421a-afb3-38bc25bd0e9c" path="/var/lib/kubelet/pods/10a2fbdf-9d64-421a-afb3-38bc25bd0e9c/volumes" Oct 30 15:06:52.310038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.309999 6554 generic.go:356] "Generic (PLEG): container finished" podID="0eceea3c-3645-4ea9-86a9-3215761db08e" containerID="492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac" exitCode=0 Oct 30 15:06:52.310226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.310042 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" Oct 30 15:06:52.310226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.310080 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" event={"ID":"0eceea3c-3645-4ea9-86a9-3215761db08e","Type":"ContainerDied","Data":"492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac"} Oct 30 15:06:52.310226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.310133 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb" event={"ID":"0eceea3c-3645-4ea9-86a9-3215761db08e","Type":"ContainerDied","Data":"a824c9d0b9449a57cbd0a615ea8c118e6132fb6ed1ed67d4c15b3e83dccc5bc8"} Oct 30 15:06:52.310226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.310147 6554 scope.go:122] "RemoveContainer" containerID="492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac" Oct 30 15:06:52.312578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.312549 6554 generic.go:356] "Generic (PLEG): container finished" podID="76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" containerID="ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae" exitCode=0 Oct 30 15:06:52.312749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.312623 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" Oct 30 15:06:52.312749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.312665 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" event={"ID":"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca","Type":"ContainerDied","Data":"ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae"} Oct 30 15:06:52.312749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.312708 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82" event={"ID":"76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca","Type":"ContainerDied","Data":"9500d62138ba2e35888a42fdea3bb8bee98a6c25347194b3ae041302df0bef1c"} Oct 30 15:06:52.321555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.321530 6554 scope.go:122] "RemoveContainer" containerID="492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac" Oct 30 15:06:52.321863 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:52.321841 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac\": container with ID starting with 492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac not found: ID does not exist" containerID="492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac" Oct 30 15:06:52.321935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.321871 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac"} err="failed to get container status \"492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac\": rpc error: code = NotFound desc = could not find container \"492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac\": container with ID starting with 492025d3f511abad3d89b739d83d88d2ff6b7829485e1f60183117d65547d7ac not found: ID does not exist" Oct 30 15:06:52.321935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.321889 6554 scope.go:122] "RemoveContainer" containerID="ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae" Oct 30 15:06:52.332267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.332235 6554 scope.go:122] "RemoveContainer" containerID="ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae" Oct 30 15:06:52.332565 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:52.332546 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae\": container with ID starting with ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae not found: ID does not exist" containerID="ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae" Oct 30 15:06:52.332632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.332573 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae"} err="failed to get container status \"ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae\": rpc error: code = NotFound desc = could not find container \"ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae\": container with ID starting with ef65abef3e99acbb243e7b3a61bef05be6c0ea7ca593c16a1d521d9c972e13ae not found: ID does not exist" Oct 30 15:06:52.370449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.370427 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82"] Oct 30 15:06:52.382170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.382135 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dc34df04-75fdcdb9d6-45n82"] Oct 30 15:06:52.466373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.461493 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb"] Oct 30 15:06:52.508374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.503565 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dc34df04-5497f599ff-kj9vb"] Oct 30 15:06:52.916001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.915964 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94"] Oct 30 15:06:52.916213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:52.916183 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" podUID="1a01e9b9-86e6-4ef3-aed6-ff07530cf464" containerName="maas-api" containerID="cri-o://59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131" gracePeriod=30 Oct 30 15:06:53.218396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.218021 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:53.334544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.334511 6554 generic.go:356] "Generic (PLEG): container finished" podID="1a01e9b9-86e6-4ef3-aed6-ff07530cf464" containerID="59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131" exitCode=0 Oct 30 15:06:53.334713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.334620 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" Oct 30 15:06:53.334713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.334620 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" event={"ID":"1a01e9b9-86e6-4ef3-aed6-ff07530cf464","Type":"ContainerDied","Data":"59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131"} Oct 30 15:06:53.334871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.334727 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94" event={"ID":"1a01e9b9-86e6-4ef3-aed6-ff07530cf464","Type":"ContainerDied","Data":"a9fcb71832aafaefd0b4f226e6fd7ef5e9fe2d90e8c78001a25a3242a4a5c15b"} Oct 30 15:06:53.334871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.334771 6554 scope.go:122] "RemoveContainer" containerID="59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131" Oct 30 15:06:53.343732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.343699 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca\") pod \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " Oct 30 15:06:53.343888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.343875 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7\" (UniqueName: \"kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7\") pod \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " Oct 30 15:06:53.343968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.343953 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\") pod \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " Oct 30 15:06:53.344076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.344057 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca" pod "1a01e9b9-86e6-4ef3-aed6-ff07530cf464" (UID: "1a01e9b9-86e6-4ef3-aed6-ff07530cf464"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:06:53.344136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.344108 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs\") pod \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\" (UID: \"1a01e9b9-86e6-4ef3-aed6-ff07530cf464\") " Oct 30 15:06:53.344800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.344780 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:53.346074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.346053 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls" pod "1a01e9b9-86e6-4ef3-aed6-ff07530cf464" (UID: "1a01e9b9-86e6-4ef3-aed6-ff07530cf464"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:53.346564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.346538 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs" pod "1a01e9b9-86e6-4ef3-aed6-ff07530cf464" (UID: "1a01e9b9-86e6-4ef3-aed6-ff07530cf464"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:06:53.346647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.346557 6554 scope.go:122] "RemoveContainer" containerID="59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131" Oct 30 15:06:53.346647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.346549 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7" pod "1a01e9b9-86e6-4ef3-aed6-ff07530cf464" (UID: "1a01e9b9-86e6-4ef3-aed6-ff07530cf464"). InnerVolumeSpecName "kube-api-access-8qqd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:06:53.346889 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:06:53.346870 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131\": container with ID starting with 59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131 not found: ID does not exist" containerID="59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131" Oct 30 15:06:53.346944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.346900 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131"} err="failed to get container status \"59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131\": rpc error: code = NotFound desc = could not find container \"59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131\": container with ID starting with 59593aa2ca6e74924b429b2406c6a8ff01811364feda67f6025d80840e079131 not found: ID does not exist" Oct 30 15:06:53.459373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.455753 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqd7\" (UniqueName: \"kubernetes.io/projected/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-kube-api-access-8qqd7\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:53.459373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.455788 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:53.459373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.455801 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a01e9b9-86e6-4ef3-aed6-ff07530cf464-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:06:53.685849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.685819 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94"] Oct 30 15:06:53.701454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:53.701421 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dc34df04-5d6dfc56b4-djc94"] Oct 30 15:06:54.168139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:54.166496 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eceea3c-3645-4ea9-86a9-3215761db08e" path="/var/lib/kubelet/pods/0eceea3c-3645-4ea9-86a9-3215761db08e/volumes" Oct 30 15:06:54.168139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:54.167148 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a01e9b9-86e6-4ef3-aed6-ff07530cf464" path="/var/lib/kubelet/pods/1a01e9b9-86e6-4ef3-aed6-ff07530cf464/volumes" Oct 30 15:06:54.168139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:06:54.167704 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" path="/var/lib/kubelet/pods/76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca/volumes" Oct 30 15:07:09.201172 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:07:09.201136 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:07:38.784123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.784085 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp"] Oct 30 15:07:38.785048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785026 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1a01e9b9-86e6-4ef3-aed6-ff07530cf464" containerName="maas-api" Oct 30 15:07:38.785134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785051 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a01e9b9-86e6-4ef3-aed6-ff07530cf464" containerName="maas-api" Oct 30 15:07:38.785134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785078 6554 cpu_manager.go:414] "containerMap: removing container" podUID="0eceea3c-3645-4ea9-86a9-3215761db08e" containerName="payload-pre-processing" Oct 30 15:07:38.785134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785089 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eceea3c-3645-4ea9-86a9-3215761db08e" containerName="payload-pre-processing" Oct 30 15:07:38.785134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785103 6554 cpu_manager.go:414] "containerMap: removing container" podUID="76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" containerName="payload-processing" Oct 30 15:07:38.785134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785110 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" containerName="payload-processing" Oct 30 15:07:38.785310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785286 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a01e9b9-86e6-4ef3-aed6-ff07530cf464" containerName="maas-api" Oct 30 15:07:38.785310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785298 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eceea3c-3645-4ea9-86a9-3215761db08e" containerName="payload-pre-processing" Oct 30 15:07:38.785310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.785308 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ffa6e8-803e-46c3-bc5a-ff5ba55e6aca" containerName="payload-processing" Oct 30 15:07:38.789482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.789465 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.810245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.810216 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp"] Oct 30 15:07:38.816155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.816126 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:07:38.816330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.816314 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:07:38.852369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.852325 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.852501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.852468 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.852637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.852618 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swqtr\" (UniqueName: \"kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.953640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.953601 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swqtr\" (UniqueName: \"kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.953844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.953824 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.953938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.953901 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.954718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.954691 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:38.956151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:38.956131 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:39.002291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:39.002263 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swqtr\" (UniqueName: \"kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr\") pod \"maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:39.112990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:39.112919 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:39.218607 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:07:39.218565 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:07:39.434052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:39.434026 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp"] Oct 30 15:07:39.434526 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:07:39.434504 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03014d8_810a_43f5_a091_f93aaed374f2.slice/crio-37d39b5739ac268709e3e2c0b4bd4f013de55940d44c69a678eb59b1fce8b4d2 WatchSource:0}: Error finding container 37d39b5739ac268709e3e2c0b4bd4f013de55940d44c69a678eb59b1fce8b4d2: Status 404 returned error can't find the container with id 37d39b5739ac268709e3e2c0b4bd4f013de55940d44c69a678eb59b1fce8b4d2 Oct 30 15:07:39.721498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:39.721417 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" event={"ID":"c03014d8-810a-43f5-a091-f93aaed374f2","Type":"ContainerStarted","Data":"37d39b5739ac268709e3e2c0b4bd4f013de55940d44c69a678eb59b1fce8b4d2"} Oct 30 15:07:40.737938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:40.737904 6554 generic.go:356] "Generic (PLEG): container finished" podID="c03014d8-810a-43f5-a091-f93aaed374f2" containerID="0806c19cfec5b224271a3f0d09be2580aa20a6653439e7661393830adb4f009b" exitCode=0 Oct 30 15:07:40.738675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:40.737985 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" event={"ID":"c03014d8-810a-43f5-a091-f93aaed374f2","Type":"ContainerDied","Data":"0806c19cfec5b224271a3f0d09be2580aa20a6653439e7661393830adb4f009b"} Oct 30 15:07:41.927660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.927567 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:41.994074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.994045 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca\") pod \"c03014d8-810a-43f5-a091-f93aaed374f2\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " Oct 30 15:07:41.994265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.994244 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr\" (UniqueName: \"kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr\") pod \"c03014d8-810a-43f5-a091-f93aaed374f2\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " Oct 30 15:07:41.994382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.994344 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca\") pod \"c03014d8-810a-43f5-a091-f93aaed374f2\" (UID: \"c03014d8-810a-43f5-a091-f93aaed374f2\") " Oct 30 15:07:41.994887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.994849 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca" pod "c03014d8-810a-43f5-a091-f93aaed374f2" (UID: "c03014d8-810a-43f5-a091-f93aaed374f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:07:41.995163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.995139 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:07:41.996769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.996692 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr" pod "c03014d8-810a-43f5-a091-f93aaed374f2" (UID: "c03014d8-810a-43f5-a091-f93aaed374f2"). InnerVolumeSpecName "kube-api-access-swqtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:07:41.997072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:41.997040 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca" pod "c03014d8-810a-43f5-a091-f93aaed374f2" (UID: "c03014d8-810a-43f5-a091-f93aaed374f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:07:42.096741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:42.096696 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c03014d8-810a-43f5-a091-f93aaed374f2-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:07:42.096741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:42.096732 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swqtr\" (UniqueName: \"kubernetes.io/projected/c03014d8-810a-43f5-a091-f93aaed374f2-kube-api-access-swqtr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:07:42.752702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:42.752668 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" event={"ID":"c03014d8-810a-43f5-a091-f93aaed374f2","Type":"ContainerDied","Data":"37d39b5739ac268709e3e2c0b4bd4f013de55940d44c69a678eb59b1fce8b4d2"} Oct 30 15:07:42.752702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:42.752698 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp" Oct 30 15:07:42.752886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:42.752728 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37d39b5739ac268709e3e2c0b4bd4f013de55940d44c69a678eb59b1fce8b4d2" Oct 30 15:07:51.555299 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.554148 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f"] Oct 30 15:07:51.559386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.559322 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c03014d8-810a-43f5-a091-f93aaed374f2" containerName="revoke-keys" Oct 30 15:07:51.559386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.559368 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03014d8-810a-43f5-a091-f93aaed374f2" containerName="revoke-keys" Oct 30 15:07:51.559674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.559574 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03014d8-810a-43f5-a091-f93aaed374f2" containerName="revoke-keys" Oct 30 15:07:51.565126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.565103 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.587557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.587520 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f"] Oct 30 15:07:51.591913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.591776 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-609709\"" Oct 30 15:07:51.591913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.591889 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-609709-dockercfg-z4qbh\"" Oct 30 15:07:51.617459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.617436 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl"] Oct 30 15:07:51.621776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.621757 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.639126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.639092 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.639341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.639288 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95h9v\" (UniqueName: \"kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v\") pod \"payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.644120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.644097 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl"] Oct 30 15:07:51.741855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.741821 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.742023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.741948 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.742098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.742030 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95h9v\" (UniqueName: \"kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v\") pod \"payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.742155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.742110 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqcds\" (UniqueName: \"kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds\") pod \"payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.742538 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.742520 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.766643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.766619 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95h9v\" (UniqueName: \"kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v\") pod \"payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.843626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.843533 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.843987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.843965 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqcds\" (UniqueName: \"kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds\") pod \"payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.864250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.845973 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.889673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.889643 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:51.898229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.898163 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqcds\" (UniqueName: \"kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds\") pod \"payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:51.938951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:51.938744 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:52.279451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.279415 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f"] Oct 30 15:07:52.279967 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:07:52.279937 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974af636_43be_47d7_b88c_c4f2b50ab696.slice/crio-8bdb266e2a063754102bed175193ea79240df9be1cd2d67d9f4289236c08a42a WatchSource:0}: Error finding container 8bdb266e2a063754102bed175193ea79240df9be1cd2d67d9f4289236c08a42a: Status 404 returned error can't find the container with id 8bdb266e2a063754102bed175193ea79240df9be1cd2d67d9f4289236c08a42a Oct 30 15:07:52.288300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.288272 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl"] Oct 30 15:07:52.290261 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:07:52.290213 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37205325_d62f_4f8e_9255_57f7a14abc89.slice/crio-b813738a1036397660478ddaf23eab58af4183f81e35e3a829ed3ff2f6bc721d WatchSource:0}: Error finding container b813738a1036397660478ddaf23eab58af4183f81e35e3a829ed3ff2f6bc721d: Status 404 returned error can't find the container with id b813738a1036397660478ddaf23eab58af4183f81e35e3a829ed3ff2f6bc721d Oct 30 15:07:52.653509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.651587 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp"] Oct 30 15:07:52.661470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.661441 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-15214ce4-2x8mp"] Oct 30 15:07:52.840162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.838760 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" event={"ID":"974af636-43be-47d7-b88c-c4f2b50ab696","Type":"ContainerStarted","Data":"9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f"} Oct 30 15:07:52.840162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.838867 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" event={"ID":"974af636-43be-47d7-b88c-c4f2b50ab696","Type":"ContainerStarted","Data":"8bdb266e2a063754102bed175193ea79240df9be1cd2d67d9f4289236c08a42a"} Oct 30 15:07:52.840162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.839205 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:07:52.840868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.840832 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" event={"ID":"37205325-d62f-4f8e-9255-57f7a14abc89","Type":"ContainerStarted","Data":"fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801"} Oct 30 15:07:52.840997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.840872 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" event={"ID":"37205325-d62f-4f8e-9255-57f7a14abc89","Type":"ContainerStarted","Data":"b813738a1036397660478ddaf23eab58af4183f81e35e3a829ed3ff2f6bc721d"} Oct 30 15:07:52.840997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.840908 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:07:52.886379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.886298 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" podStartSLOduration=1.886280256 podStartE2EDuration="1.886280256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:52.884008416 +0000 UTC m=+1176.843049820" watchObservedRunningTime="2026-10-30 15:07:52.886280256 +0000 UTC m=+1176.845321661" Oct 30 15:07:52.930699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:52.930574 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" podStartSLOduration=1.9305591039999999 podStartE2EDuration="1.930559104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:07:52.927801988 +0000 UTC m=+1176.886843394" watchObservedRunningTime="2026-10-30 15:07:52.930559104 +0000 UTC m=+1176.889600529" Oct 30 15:07:54.155017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:07:54.154985 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03014d8-810a-43f5-a091-f93aaed374f2" path="/var/lib/kubelet/pods/c03014d8-810a-43f5-a091-f93aaed374f2/volumes" Oct 30 15:08:01.707691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.707601 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz"] Oct 30 15:08:01.712772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.712742 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.732796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.732759 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz"] Oct 30 15:08:01.741033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.741006 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-416d92\"" Oct 30 15:08:01.779574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.779546 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw"] Oct 30 15:08:01.784179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.784158 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:01.796340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.796314 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:01.796722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.796694 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.796826 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.796785 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvwv\" (UniqueName: \"kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv\") pod \"payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.796891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.796865 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9p6\" (UniqueName: \"kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6\") pod \"payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:01.800696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.800678 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-416d92-dockercfg-stw2x\"" Oct 30 15:08:01.801549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.801528 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw"] Oct 30 15:08:01.928374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.911475 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.931742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.931714 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvwv\" (UniqueName: \"kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv\") pod \"payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.932339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.932315 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9p6\" (UniqueName: \"kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6\") pod \"payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:01.975278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.962539 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:01.975278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.963783 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:01.975278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.931336 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.975278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.972983 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvwv\" (UniqueName: \"kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv\") pod \"payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:01.985343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:01.985319 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9p6\" (UniqueName: \"kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6\") pod \"payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:02.029213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:02.029177 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:02.104421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:02.104376 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:02.375895 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:02.375854 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd409ed2_dcde_47cf_9deb_59ba2121ebed.slice/crio-9af91660c5a0e406c633ba696cb190b14baa8d6ff61bdc70144ea25baaa96919 WatchSource:0}: Error finding container 9af91660c5a0e406c633ba696cb190b14baa8d6ff61bdc70144ea25baaa96919: Status 404 returned error can't find the container with id 9af91660c5a0e406c633ba696cb190b14baa8d6ff61bdc70144ea25baaa96919 Oct 30 15:08:02.377989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:02.377960 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz"] Oct 30 15:08:02.449990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:02.449965 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw"] Oct 30 15:08:02.450606 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:02.450579 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a541bd_a2fd_4488_8695_3de8442c4438.slice/crio-0fd0453c6ff37cd5f7bbb38a47c364b6f00761c944ecdb407f2a866641ed0e3a WatchSource:0}: Error finding container 0fd0453c6ff37cd5f7bbb38a47c364b6f00761c944ecdb407f2a866641ed0e3a: Status 404 returned error can't find the container with id 0fd0453c6ff37cd5f7bbb38a47c364b6f00761c944ecdb407f2a866641ed0e3a Oct 30 15:08:03.014298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.014256 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" event={"ID":"dd409ed2-dcde-47cf-9deb-59ba2121ebed","Type":"ContainerStarted","Data":"2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc"} Oct 30 15:08:03.014854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.014326 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" event={"ID":"dd409ed2-dcde-47cf-9deb-59ba2121ebed","Type":"ContainerStarted","Data":"9af91660c5a0e406c633ba696cb190b14baa8d6ff61bdc70144ea25baaa96919"} Oct 30 15:08:03.014854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.014439 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:03.016153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.016123 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" event={"ID":"27a541bd-a2fd-4488-8695-3de8442c4438","Type":"ContainerStarted","Data":"3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26"} Oct 30 15:08:03.016276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.016169 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" event={"ID":"27a541bd-a2fd-4488-8695-3de8442c4438","Type":"ContainerStarted","Data":"0fd0453c6ff37cd5f7bbb38a47c364b6f00761c944ecdb407f2a866641ed0e3a"} Oct 30 15:08:03.016276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.016197 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:03.070643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.070578 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" podStartSLOduration=2.070556835 podStartE2EDuration="2.070556835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:03.067574299 +0000 UTC m=+1187.026615707" watchObservedRunningTime="2026-10-30 15:08:03.070556835 +0000 UTC m=+1187.029598241" Oct 30 15:08:03.115103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.115037 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" podStartSLOduration=2.115020287 podStartE2EDuration="2.115020287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:03.113650157 +0000 UTC m=+1187.072691564" watchObservedRunningTime="2026-10-30 15:08:03.115020287 +0000 UTC m=+1187.074061693" Oct 30 15:08:03.850099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.850071 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:08:03.850274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:03.850142 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:08:09.212603 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:08:09.212563 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:08:14.032209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:14.032170 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:08:14.033428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:14.033403 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:08:16.617486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:16.617456 6554 scope.go:122] "RemoveContainer" containerID="42f0f7735d99ec9a8e3f75fc3b7f0fb266b8442479f2adc7b9be9c58092951b9" Oct 30 15:08:16.627020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:16.626998 6554 scope.go:122] "RemoveContainer" containerID="95f71394717eb41a7bc676f20b1542761b63c02422f17c61cc119bd85644db4d" Oct 30 15:08:16.635817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:16.635797 6554 scope.go:122] "RemoveContainer" containerID="48ffcab7186d0ea46ef99a7cc7cf6dfd09fb1fc58e6aef286241d48be4720663" Oct 30 15:08:22.375927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.375890 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2"] Oct 30 15:08:22.381681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.381651 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.390584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.390473 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww"] Oct 30 15:08:22.395828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.395805 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.408879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.408736 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2"] Oct 30 15:08:22.422570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.422542 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww"] Oct 30 15:08:22.473894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.473869 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29rn\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.473906 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.473925 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474072 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpq9k\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474109 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474191 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474258 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474289 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474380 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474400 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474425 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474441 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474496 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474554 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474574 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.474644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474590 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474611 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.474709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.474689 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.576362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576326 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpq9k\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576390 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.576479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576420 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.576479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576471 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.576597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576510 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576554 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.576597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576582 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576617 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576640 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576669 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576714 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.576744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576738 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576765 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576797 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576832 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576830 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576924 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576956 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29rn\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.576990 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.577385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577012 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577030 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.577385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577218 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577423 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.577556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577432 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.577556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577453 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.577923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.577898 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.578086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.578064 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.578171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.578094 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.579059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.579037 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.579499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.579482 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.579560 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.579478 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.579641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.579624 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.602452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.602426 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29rn\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.602527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.602470 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpq9k\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.602527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.602492 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-l99j2\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.602986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.602970 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-k9tww\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:22.706293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.706274 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:22.718784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:22.718760 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:23.050455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.050422 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww"] Oct 30 15:08:23.050569 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:23.050475 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d9982e_1bda_4194_9c76_20258adc90b9.slice/crio-ffa4a07a83537fb582421a5005a402a6cc64b59a3ead47d40b0bf98dcd4cd71c WatchSource:0}: Error finding container ffa4a07a83537fb582421a5005a402a6cc64b59a3ead47d40b0bf98dcd4cd71c: Status 404 returned error can't find the container with id ffa4a07a83537fb582421a5005a402a6cc64b59a3ead47d40b0bf98dcd4cd71c Oct 30 15:08:23.053196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.053162 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:08:23.053270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.053242 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:08:23.053312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.053284 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:08:23.054584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.054563 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2"] Oct 30 15:08:23.067542 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:23.067521 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5af2bfc_e804_4826_9488_61959cfff44e.slice/crio-20da19bececbcde089907223002978a69bb3163363b0a97209aeee3c5e5bd24e WatchSource:0}: Error finding container 20da19bececbcde089907223002978a69bb3163363b0a97209aeee3c5e5bd24e: Status 404 returned error can't find the container with id 20da19bececbcde089907223002978a69bb3163363b0a97209aeee3c5e5bd24e Oct 30 15:08:23.069889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.069857 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:08:23.069989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.069919 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:08:23.069989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.069948 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:08:23.200707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.200661 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" event={"ID":"e5af2bfc-e804-4826-9488-61959cfff44e","Type":"ContainerStarted","Data":"4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc"} Oct 30 15:08:23.200903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.200725 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" event={"ID":"e5af2bfc-e804-4826-9488-61959cfff44e","Type":"ContainerStarted","Data":"20da19bececbcde089907223002978a69bb3163363b0a97209aeee3c5e5bd24e"} Oct 30 15:08:23.203860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.203817 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" event={"ID":"a5d9982e-1bda-4194-9c76-20258adc90b9","Type":"ContainerStarted","Data":"886f42ec21774dc585796d7bb193ba7ce16c25c26d7b72e250f42e2b2134867e"} Oct 30 15:08:23.204016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.203875 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" event={"ID":"a5d9982e-1bda-4194-9c76-20258adc90b9","Type":"ContainerStarted","Data":"ffa4a07a83537fb582421a5005a402a6cc64b59a3ead47d40b0bf98dcd4cd71c"} Oct 30 15:08:23.289187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.289138 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podStartSLOduration=1.289125124 podStartE2EDuration="1.289125124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:23.286887797 +0000 UTC m=+1207.245929204" watchObservedRunningTime="2026-10-30 15:08:23.289125124 +0000 UTC m=+1207.248166529" Oct 30 15:08:23.707089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.707055 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:23.719199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:23.719170 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:24.708413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:24.708179 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:24.708413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:24.708226 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:24.720056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:24.720023 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:24.720197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:24.720105 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:25.227913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.227860 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podStartSLOduration=3.227847803 podStartE2EDuration="3.227847803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:23.345053833 +0000 UTC m=+1207.304095237" watchObservedRunningTime="2026-10-30 15:08:25.227847803 +0000 UTC m=+1209.186889216" Oct 30 15:08:25.231543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.231519 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497"] Oct 30 15:08:25.235696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.235682 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.272599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.272564 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497"] Oct 30 15:08:25.277377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.276667 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f7acff45\"" Oct 30 15:08:25.277377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.276757 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f7acff45-dockercfg-gcj58\"" Oct 30 15:08:25.471370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.470334 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr"] Oct 30 15:08:25.528441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.527679 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.528441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.527913 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr9jd\" (UniqueName: \"kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.638376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.637770 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr9jd\" (UniqueName: \"kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.638376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.638042 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.643134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.643106 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.644918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.644859 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr"] Oct 30 15:08:25.645109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.645095 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:25.719399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.719364 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr9jd\" (UniqueName: \"kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.721694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.721287 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:25.721694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.721337 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:25.726327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.726296 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:25.726486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.726377 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:25.738994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.738969 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:25.739115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.739065 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:25.840242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.840158 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:25.840400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.840370 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:25.841031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.841011 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:25.850678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.850659 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:25.906810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:25.906770 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:26.036873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.036844 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:26.307683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.307638 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497"] Oct 30 15:08:26.510726 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.510255 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr"] Oct 30 15:08:26.513160 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:26.512975 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fee4b6a_11e1_46e6_8636_4626b57c203c.slice/crio-93902765405c57c876e0dcbdd1b4c28ef3880257f3aecc02952fb545e1146e48 WatchSource:0}: Error finding container 93902765405c57c876e0dcbdd1b4c28ef3880257f3aecc02952fb545e1146e48: Status 404 returned error can't find the container with id 93902765405c57c876e0dcbdd1b4c28ef3880257f3aecc02952fb545e1146e48 Oct 30 15:08:26.707809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.707718 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:26.707809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.707772 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:26.719896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.719680 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:26.719896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:26.719725 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:27.031045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.030968 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6"] Oct 30 15:08:27.044496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.044465 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1fd2c76-zmdc6"] Oct 30 15:08:27.243224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.243192 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" event={"ID":"3fee4b6a-11e1-46e6-8636-4626b57c203c","Type":"ContainerStarted","Data":"f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02"} Oct 30 15:08:27.243427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.243246 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" event={"ID":"3fee4b6a-11e1-46e6-8636-4626b57c203c","Type":"ContainerStarted","Data":"93902765405c57c876e0dcbdd1b4c28ef3880257f3aecc02952fb545e1146e48"} Oct 30 15:08:27.243427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.243290 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:27.244858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.244831 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" event={"ID":"69b10318-ff5f-41cf-bcd2-ed79fdfd526a","Type":"ContainerStarted","Data":"0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f"} Oct 30 15:08:27.244966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.244880 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" event={"ID":"69b10318-ff5f-41cf-bcd2-ed79fdfd526a","Type":"ContainerStarted","Data":"4454b645766019531a4a569b21df1bd9ac0cb7cc2b1284fce562dcf4b7607344"} Oct 30 15:08:27.245133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.245120 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:27.309809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.309066 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" podStartSLOduration=2.309047402 podStartE2EDuration="2.309047402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:27.303160963 +0000 UTC m=+1211.262202369" watchObservedRunningTime="2026-10-30 15:08:27.309047402 +0000 UTC m=+1211.268088809" Oct 30 15:08:27.349590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.349534 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" podStartSLOduration=2.349520669 podStartE2EDuration="2.349520669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:27.347016223 +0000 UTC m=+1211.306057629" watchObservedRunningTime="2026-10-30 15:08:27.349520669 +0000 UTC m=+1211.308562074" Oct 30 15:08:27.707565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.706810 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:27.707565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.706893 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:27.719763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.719734 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:27.719899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:27.719782 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:28.168442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:28.160728 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3222be-241e-4297-9aae-f2b3bb95cdb9" path="/var/lib/kubelet/pods/7a3222be-241e-4297-9aae-f2b3bb95cdb9/volumes" Oct 30 15:08:28.707791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:28.707757 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:28.707982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:28.707939 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:28.719475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:28.719437 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:28.719615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:28.719479 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:29.707776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:29.707738 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:29.708405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:29.708370 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:29.723656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:29.722694 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:29.723656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:29.722763 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:30.706868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:30.706825 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:30.707067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:30.706891 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:30.719477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:30.719448 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 15:08:30.719939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:30.719907 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 15:08:31.706771 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:31.706728 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 15:08:31.707076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:31.707041 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 15:08:32.774373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:32.774059 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:08:32.774373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:32.774137 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:08:33.156374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:33.155554 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:33.156752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:33.156716 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:33.157736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:33.157712 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:08:33.158197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:33.158177 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:33.159655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:33.159614 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:33.163629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:33.163502 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:08:38.255872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:38.255842 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:08:38.256255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:38.255903 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:08:39.202443 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:08:39.202409 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:08:40.030264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:40.030225 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg"] Oct 30 15:08:40.042853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:40.042821 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f95e2309-wccsg"] Oct 30 15:08:40.161730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:40.161691 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa05b0d6-0312-4a3b-b1f6-00d4433ae495" path="/var/lib/kubelet/pods/aa05b0d6-0312-4a3b-b1f6-00d4433ae495/volumes" Oct 30 15:08:45.768565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.768535 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2"] Oct 30 15:08:45.775014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.774993 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:45.799734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.799710 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2"] Oct 30 15:08:45.807217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.807186 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d57911\"" Oct 30 15:08:45.807365 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.807287 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d57911-dockercfg-rgrlj\"" Oct 30 15:08:45.852116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.852086 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw"] Oct 30 15:08:45.857005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.856981 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:45.857873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.857841 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:45.858038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.858012 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcdk\" (UniqueName: \"kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk\") pod \"payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:45.873930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.873901 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw"] Oct 30 15:08:46.000373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.987414 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z258\" (UniqueName: \"kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258\") pod \"payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.000373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.987631 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:46.000373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.987788 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcdk\" (UniqueName: \"kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk\") pod \"payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:46.000373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.987854 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.000373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:45.989014 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:46.020889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.020817 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcdk\" (UniqueName: \"kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk\") pod \"payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:46.089529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.089497 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.089706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.089600 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z258\" (UniqueName: \"kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258\") pod \"payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.090262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.090243 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.090461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.090448 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:46.119830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.119795 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z258\" (UniqueName: \"kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258\") pod \"payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.176041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.175807 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:46.425149 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:46.425116 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9864e8_ab83_406e_b412_f88427a3671d.slice/crio-788bd7ff6bfdcb91828ba57bd5adbf7426c55210d970670c6bc3e59966bff96c WatchSource:0}: Error finding container 788bd7ff6bfdcb91828ba57bd5adbf7426c55210d970670c6bc3e59966bff96c: Status 404 returned error can't find the container with id 788bd7ff6bfdcb91828ba57bd5adbf7426c55210d970670c6bc3e59966bff96c Oct 30 15:08:46.425303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.425284 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2"] Oct 30 15:08:46.478225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.478193 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" event={"ID":"ac9864e8-ab83-406e-b412-f88427a3671d","Type":"ContainerStarted","Data":"788bd7ff6bfdcb91828ba57bd5adbf7426c55210d970670c6bc3e59966bff96c"} Oct 30 15:08:46.549530 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:46.549503 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e63662_dd11_41ba_be9c_48db54705e50.slice/crio-ef0106738cd7b2c924d6e51304766f8a32897609423f6cf761795f5adb69bbc9 WatchSource:0}: Error finding container ef0106738cd7b2c924d6e51304766f8a32897609423f6cf761795f5adb69bbc9: Status 404 returned error can't find the container with id ef0106738cd7b2c924d6e51304766f8a32897609423f6cf761795f5adb69bbc9 Oct 30 15:08:46.550048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:46.550027 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw"] Oct 30 15:08:47.489874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.489837 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" event={"ID":"ac9864e8-ab83-406e-b412-f88427a3671d","Type":"ContainerStarted","Data":"51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d"} Oct 30 15:08:47.490307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.489937 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:08:47.491837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.491808 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" event={"ID":"60e63662-dd11-41ba-be9c-48db54705e50","Type":"ContainerStarted","Data":"a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085"} Oct 30 15:08:47.491994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.491862 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" event={"ID":"60e63662-dd11-41ba-be9c-48db54705e50","Type":"ContainerStarted","Data":"ef0106738cd7b2c924d6e51304766f8a32897609423f6cf761795f5adb69bbc9"} Oct 30 15:08:47.491994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.491903 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:47.547598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.547528 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" podStartSLOduration=2.547510463 podStartE2EDuration="2.547510463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:47.545563464 +0000 UTC m=+1231.504604869" watchObservedRunningTime="2026-10-30 15:08:47.547510463 +0000 UTC m=+1231.506551870" Oct 30 15:08:47.591817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:47.591761 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" podStartSLOduration=2.591746023 podStartE2EDuration="2.591746023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:47.589135161 +0000 UTC m=+1231.548176568" watchObservedRunningTime="2026-10-30 15:08:47.591746023 +0000 UTC m=+1231.550787427" Oct 30 15:08:48.363666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.363634 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh"] Oct 30 15:08:48.368229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.368211 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.389260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.389233 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh"] Oct 30 15:08:48.397917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.397894 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f7acff45-serving-cert\"" Oct 30 15:08:48.446259 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.446230 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.446407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.446264 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfdlj\" (UniqueName: \"kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.446503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.446477 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.446631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.446609 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.547776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.547747 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.548201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.547785 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfdlj\" (UniqueName: \"kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.548201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.547888 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.548201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.547958 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.548201 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:08:48.548053 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-serving-cert: secret "maas-api-e2e-mt-f7acff45-serving-cert" not found Oct 30 15:08:48.548201 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:08:48.548133 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls podName:4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa nodeName:}" failed. No retries permitted until 2026-10-30 15:08:49.048113139 +0000 UTC m=+1233.007154531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls") pod "maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" (UID: "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa") : secret "maas-api-e2e-mt-f7acff45-serving-cert" not found Oct 30 15:08:48.548520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.548494 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.550670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.550650 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:48.574972 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:48.574950 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfdlj\" (UniqueName: \"kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:49.057959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.057927 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:49.058143 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:08:49.058083 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-serving-cert: secret "maas-api-e2e-mt-f7acff45-serving-cert" not found Oct 30 15:08:49.058184 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:08:49.058145 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls podName:4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa nodeName:}" failed. No retries permitted until 2026-10-30 15:08:50.05813141 +0000 UTC m=+1234.017172798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls") pod "maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" (UID: "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa") : secret "maas-api-e2e-mt-f7acff45-serving-cert" not found Oct 30 15:08:49.832370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.831374 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d"] Oct 30 15:08:49.891375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.887746 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:49.932372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.918824 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-09e8eb\"" Oct 30 15:08:49.937414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.934503 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-09e8eb-dockercfg-dqcqv\"" Oct 30 15:08:49.937414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.936417 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d"] Oct 30 15:08:49.987490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.987456 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xllcj\" (UniqueName: \"kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj\") pod \"payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:49.987693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:49.987675 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:50.089115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.089030 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:50.089270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.089171 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xllcj\" (UniqueName: \"kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj\") pod \"payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:50.089433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.089413 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:50.089776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.089758 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:50.092045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.092018 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") pod \"maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:50.114214 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.114182 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xllcj\" (UniqueName: \"kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj\") pod \"payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:50.186022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.185992 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:50.298083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.298043 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:50.523973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.523852 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh"] Oct 30 15:08:50.660342 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:08:50.660310 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce158e2f_0d04_49f3_a407_5ad3f3602383.slice/crio-f6046da02056ac175d1729929713e53653a6f4415e7ce4debaafd163f892412a WatchSource:0}: Error finding container f6046da02056ac175d1729929713e53653a6f4415e7ce4debaafd163f892412a: Status 404 returned error can't find the container with id f6046da02056ac175d1729929713e53653a6f4415e7ce4debaafd163f892412a Oct 30 15:08:50.661072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:50.661025 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d"] Oct 30 15:08:51.543618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.543582 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" event={"ID":"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa","Type":"ContainerStarted","Data":"bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8"} Oct 30 15:08:51.544128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.543646 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:51.544128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.543669 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" event={"ID":"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa","Type":"ContainerStarted","Data":"05f060191986d71370aee77fa7748cdc1f108eba88f4cf15edbbfcaa9603ab6b"} Oct 30 15:08:51.545430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.545400 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" event={"ID":"ce158e2f-0d04-49f3-a407-5ad3f3602383","Type":"ContainerStarted","Data":"43ff541d8225bc24b5eeddf2d515cf2ad4044dc295b37ff8284cbcb5a9d97e7d"} Oct 30 15:08:51.545546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.545452 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" event={"ID":"ce158e2f-0d04-49f3-a407-5ad3f3602383","Type":"ContainerStarted","Data":"f6046da02056ac175d1729929713e53653a6f4415e7ce4debaafd163f892412a"} Oct 30 15:08:51.545546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.545503 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:08:51.593066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.593018 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" podStartSLOduration=3.263831209 podStartE2EDuration="3.593005617s" totalImagesPullingTime="329.174408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:51.589151331 +0000 UTC m=+1235.548192732" watchObservedRunningTime="2026-10-30 15:08:51.593005617 +0000 UTC m=+1235.552047021" Oct 30 15:08:51.635092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:51.635045 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" podStartSLOduration=2.635035251 podStartE2EDuration="2.635035251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:08:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:08:51.63179952 +0000 UTC m=+1235.590840925" watchObservedRunningTime="2026-10-30 15:08:51.635035251 +0000 UTC m=+1235.594076650" Oct 30 15:08:53.031421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:53.031386 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg"] Oct 30 15:08:53.044442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:53.044415 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5547ad9f-mz8xg"] Oct 30 15:08:54.156420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:54.156389 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5f40f7-63e1-45ad-b776-57374f833d59" path="/var/lib/kubelet/pods/ee5f40f7-63e1-45ad-b776-57374f833d59/volumes" Oct 30 15:08:57.557937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:57.557912 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:08:58.501609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:58.501583 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:08:58.501783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:08:58.501645 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:09:02.556424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:02.556392 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:09:03.831574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:03.828432 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-2cfhn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:09:03.831574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:03.828508 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:04.210373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:04.203657 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-k9tww container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:09:04.210373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:04.203714 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:04.286372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:04.267332 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-l99j2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:09:04.286372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:04.267438 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:04.604725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:04.604619 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:09.206780 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:09:09.206721 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:09:10.030044 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:10.030013 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45"] Oct 30 15:09:10.043745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:10.043718 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e6bd936-b9r45"] Oct 30 15:09:10.156356 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:10.156326 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36a6b811-11d0-4712-b551-37a74a2b96a3" path="/var/lib/kubelet/pods/36a6b811-11d0-4712-b551-37a74a2b96a3/volumes" Oct 30 15:09:12.691508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.691477 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l"] Oct 30 15:09:12.696072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.696042 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.715053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.715025 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-d57911-serving-cert\"" Oct 30 15:09:12.733668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.733642 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l"] Oct 30 15:09:12.773635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.773594 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.773776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.773643 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t94h\" (UniqueName: \"kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.773776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.773752 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.773979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.773954 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.875202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.875172 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.875332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.875296 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.875332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.875316 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t94h\" (UniqueName: \"kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.875499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.875385 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.875963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.875941 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.877898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.877879 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.878025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.878007 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:12.900108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:12.900086 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t94h\" (UniqueName: \"kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h\") pod \"maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:13.010825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:13.010746 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:13.335216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:13.335185 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l"] Oct 30 15:09:13.335361 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:09:13.335288 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1b51cc_fb11_4ccc_81d4_3a0de4d8d471.slice/crio-03c3f1950b6733530e30c5dce68b2fdbe3bcb7be6e925764fc79a82e0b2a6fb7 WatchSource:0}: Error finding container 03c3f1950b6733530e30c5dce68b2fdbe3bcb7be6e925764fc79a82e0b2a6fb7: Status 404 returned error can't find the container with id 03c3f1950b6733530e30c5dce68b2fdbe3bcb7be6e925764fc79a82e0b2a6fb7 Oct 30 15:09:13.847638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:13.847601 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" event={"ID":"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471","Type":"ContainerStarted","Data":"d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda"} Oct 30 15:09:13.848098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:13.847682 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" event={"ID":"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471","Type":"ContainerStarted","Data":"03c3f1950b6733530e30c5dce68b2fdbe3bcb7be6e925764fc79a82e0b2a6fb7"} Oct 30 15:09:13.848098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:13.847867 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:13.897008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:13.896932 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" podStartSLOduration=1.5952865470000002 podStartE2EDuration="1.896919249s" totalImagesPullingTime="301.632702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:13.892941359 +0000 UTC m=+1257.851982764" watchObservedRunningTime="2026-10-30 15:09:13.896919249 +0000 UTC m=+1257.855960653" Oct 30 15:09:16.666193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.666153 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn"] Oct 30 15:09:16.672608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.672579 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.692416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.692387 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn"] Oct 30 15:09:16.701516 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.701451 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-09e8eb-serving-cert\"" Oct 30 15:09:16.726329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.726307 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.726476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.726344 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.726588 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.726574 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.726672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.726656 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcfz\" (UniqueName: \"kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.827815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.827784 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.827955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.827831 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.828010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.827959 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.828066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.828048 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcfz\" (UniqueName: \"kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.828728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.828707 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.830534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.830510 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.830733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.830564 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.853557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.853530 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcfz\" (UniqueName: \"kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz\") pod \"maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:16.989059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:16.988972 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:17.031318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.031286 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97"] Oct 30 15:09:17.036659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.036632 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f8f51ca-tfb97"] Oct 30 15:09:17.333551 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.333525 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn"] Oct 30 15:09:17.333859 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:09:17.333837 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efe748d_06f9_4d43_8de8_241ae8b2f04e.slice/crio-333af12429f3db0c542f5960b93d8f29f1cfdf434a6f190e01766f130468f294 WatchSource:0}: Error finding container 333af12429f3db0c542f5960b93d8f29f1cfdf434a6f190e01766f130468f294: Status 404 returned error can't find the container with id 333af12429f3db0c542f5960b93d8f29f1cfdf434a6f190e01766f130468f294 Oct 30 15:09:17.894551 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.894516 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" event={"ID":"7efe748d-06f9-4d43-8de8-241ae8b2f04e","Type":"ContainerStarted","Data":"186eef5e49d7aeabb5b2290e145d70d5de470bb108b4d5ba25fa685b8b171c78"} Oct 30 15:09:17.894909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.894584 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:17.894909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.894611 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" event={"ID":"7efe748d-06f9-4d43-8de8-241ae8b2f04e","Type":"ContainerStarted","Data":"333af12429f3db0c542f5960b93d8f29f1cfdf434a6f190e01766f130468f294"} Oct 30 15:09:17.941421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:17.941370 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" podStartSLOduration=1.507214253 podStartE2EDuration="1.941344263s" totalImagesPullingTime="434.13001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:17.938667883 +0000 UTC m=+1261.897709289" watchObservedRunningTime="2026-10-30 15:09:17.941344263 +0000 UTC m=+1261.900385669" Oct 30 15:09:18.157706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:18.157671 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f" path="/var/lib/kubelet/pods/3400b3d8-d4fe-4e35-a06b-d1d9147c5b1f/volumes" Oct 30 15:09:19.860535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:19.860503 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:09:23.910467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:23.910433 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:09:26.106009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:26.105974 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9"] Oct 30 15:09:26.121176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:26.121144 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6f0a7cb-jq5k9"] Oct 30 15:09:26.159252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:26.159221 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd626d53-f2d5-472e-8ab4-aca0a8e8923d" path="/var/lib/kubelet/pods/bd626d53-f2d5-472e-8ab4-aca0a8e8923d/volumes" Oct 30 15:09:31.322544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:31.318243 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:31.625427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:31.617097 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:31.766671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:31.763773 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:31.826373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:31.813881 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 15:09:31.918429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:31.885717 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 15:09:31.918429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:31.887192 6554 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh" podUID="72284b4e-d5b0-4317-b86f-7341f3d3364f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:32.284746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:32.284573 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:32.632668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:32.631434 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:09:39.210370 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:09:39.210134 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:09:40.044319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:40.044282 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq"] Oct 30 15:09:40.058166 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:40.058136 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4534535b-kn2kq"] Oct 30 15:09:40.155888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:40.155855 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0e3e66-c0bf-42e5-a786-3f78e5bb7869" path="/var/lib/kubelet/pods/5b0e3e66-c0bf-42e5-a786-3f78e5bb7869/volumes" Oct 30 15:09:58.232567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.232532 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg"] Oct 30 15:09:58.239694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.239665 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.255398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.255335 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg"] Oct 30 15:09:58.256458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.256433 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-7424dbac-serving-cert\"" Oct 30 15:09:58.291997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.291967 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.292174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.292157 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf75h\" (UniqueName: \"kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.292248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.292231 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.292559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.292524 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.394123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.394096 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.394278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.394146 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.394278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.394238 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf75h\" (UniqueName: \"kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.394430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.394295 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.394916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.394892 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.396567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.396547 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.396659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.396627 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.418776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.418746 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf75h\" (UniqueName: \"kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h\") pod \"maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.462322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.462278 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg"] Oct 30 15:09:58.468586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.468531 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.498720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.498676 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.498886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.498867 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwrm\" (UniqueName: \"kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm\") pod \"payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.513739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.513713 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-76473c-dockercfg-k5pz5\"" Oct 30 15:09:58.513933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.513743 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-76473c\"" Oct 30 15:09:58.554511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.554478 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg"] Oct 30 15:09:58.569920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.569643 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:58.604956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.604911 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.605327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.605301 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwrm\" (UniqueName: \"kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm\") pod \"payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.606505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.606485 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.662201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.660704 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwrm\" (UniqueName: \"kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm\") pod \"payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.897539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.897510 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:58.971836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.971744 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg"] Oct 30 15:09:58.976728 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:09:58.976689 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0316a266_070e_4bdb_875b_86383902d4e5.slice/crio-edca2d21ea1c94f5bac81d305cf52b89c19cde33029af3bdf2567dae2c82c235 WatchSource:0}: Error finding container edca2d21ea1c94f5bac81d305cf52b89c19cde33029af3bdf2567dae2c82c235: Status 404 returned error can't find the container with id edca2d21ea1c94f5bac81d305cf52b89c19cde33029af3bdf2567dae2c82c235 Oct 30 15:09:58.981386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:58.979800 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:09:59.210426 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:09:59.210394 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4afb49d4_4ab0_45c2_8fdc_190326fbcb39.slice/crio-33c4e53ef9c6e82aaa82ad92f56f7f0c39d96ff3faeacbd3d5a5d63838dfe380 WatchSource:0}: Error finding container 33c4e53ef9c6e82aaa82ad92f56f7f0c39d96ff3faeacbd3d5a5d63838dfe380: Status 404 returned error can't find the container with id 33c4e53ef9c6e82aaa82ad92f56f7f0c39d96ff3faeacbd3d5a5d63838dfe380 Oct 30 15:09:59.211467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.211446 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg"] Oct 30 15:09:59.484984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.484880 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" event={"ID":"0316a266-070e-4bdb-875b-86383902d4e5","Type":"ContainerStarted","Data":"3203948538b3d026c68b298698eeae17e3fbc336e9843834da197a7e4de03160"} Oct 30 15:09:59.484984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.484940 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" event={"ID":"0316a266-070e-4bdb-875b-86383902d4e5","Type":"ContainerStarted","Data":"edca2d21ea1c94f5bac81d305cf52b89c19cde33029af3bdf2567dae2c82c235"} Oct 30 15:09:59.484984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.484974 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:09:59.487263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.487237 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" event={"ID":"4afb49d4-4ab0-45c2-8fdc-190326fbcb39","Type":"ContainerStarted","Data":"6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866"} Oct 30 15:09:59.487446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.487281 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" event={"ID":"4afb49d4-4ab0-45c2-8fdc-190326fbcb39","Type":"ContainerStarted","Data":"33c4e53ef9c6e82aaa82ad92f56f7f0c39d96ff3faeacbd3d5a5d63838dfe380"} Oct 30 15:09:59.487446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.487306 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:09:59.536690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.536610 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" podStartSLOduration=1.215139304 podStartE2EDuration="1.536589274s" totalImagesPullingTime="321.44997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:59.531495071 +0000 UTC m=+1303.490536474" watchObservedRunningTime="2026-10-30 15:09:59.536589274 +0000 UTC m=+1303.495630679" Oct 30 15:09:59.581584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:09:59.581516 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" podStartSLOduration=1.581501348 podStartE2EDuration="1.581501348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:09:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:09:59.57863147 +0000 UTC m=+1303.537672879" watchObservedRunningTime="2026-10-30 15:09:59.581501348 +0000 UTC m=+1303.540542792" Oct 30 15:10:05.499715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.499687 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:10:05.516104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.516077 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg"] Oct 30 15:10:05.520769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.520753 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.537677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.537653 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-97ead4\"" Oct 30 15:10:05.540117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.540092 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-97ead4-dockercfg-lxwp5\"" Oct 30 15:10:05.541479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.541457 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg"] Oct 30 15:10:05.606279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.606246 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7"] Oct 30 15:10:05.612016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.611994 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.613999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.613975 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.614541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.614394 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z284n\" (UniqueName: \"kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n\") pod \"payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.629639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.629614 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7"] Oct 30 15:10:05.715722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.715689 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt77c\" (UniqueName: \"kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c\") pod \"payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.715931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.715909 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z284n\" (UniqueName: \"kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n\") pod \"payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.716002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.715945 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.716060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.716021 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.716883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.716858 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.763414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.760409 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z284n\" (UniqueName: \"kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n\") pod \"payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.827940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.825436 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.827940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.825718 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt77c\" (UniqueName: \"kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c\") pod \"payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.827940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.826598 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.841100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.840986 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:05.851136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.851110 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt77c\" (UniqueName: \"kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c\") pod \"payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:05.931417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:05.931387 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:06.190990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.190956 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg"] Oct 30 15:10:06.194824 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:10:06.194785 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73043333_745a_4a8a_a431_6f4b493d1600.slice/crio-87ad4fe724cc13814205bd1cf3a1ddee3395e149344e94b99e9284b7ece8d629 WatchSource:0}: Error finding container 87ad4fe724cc13814205bd1cf3a1ddee3395e149344e94b99e9284b7ece8d629: Status 404 returned error can't find the container with id 87ad4fe724cc13814205bd1cf3a1ddee3395e149344e94b99e9284b7ece8d629 Oct 30 15:10:06.302249 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:10:06.302220 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd8f9b8_62f3_4055_836f_6808267ab03d.slice/crio-fe5869d7c53ede34b3a236480cc86beba82cb4a4489bebafb53ddee8e1badc0a WatchSource:0}: Error finding container fe5869d7c53ede34b3a236480cc86beba82cb4a4489bebafb53ddee8e1badc0a: Status 404 returned error can't find the container with id fe5869d7c53ede34b3a236480cc86beba82cb4a4489bebafb53ddee8e1badc0a Oct 30 15:10:06.305672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.305643 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7"] Oct 30 15:10:06.545900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.545854 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" event={"ID":"73043333-745a-4a8a-a431-6f4b493d1600","Type":"ContainerStarted","Data":"26fa22b98700e80f16984a18548f88db3cd309f4c9d8ff0d3572c5df38757202"} Oct 30 15:10:06.546403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.545917 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" event={"ID":"73043333-745a-4a8a-a431-6f4b493d1600","Type":"ContainerStarted","Data":"87ad4fe724cc13814205bd1cf3a1ddee3395e149344e94b99e9284b7ece8d629"} Oct 30 15:10:06.546403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.545949 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:06.547986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.547919 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" event={"ID":"dfd8f9b8-62f3-4055-836f-6808267ab03d","Type":"ContainerStarted","Data":"a86380c8fe4c87ff48a3f3764e798c8e32e988869f5db32f84df12438ff6957c"} Oct 30 15:10:06.547986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.547959 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" event={"ID":"dfd8f9b8-62f3-4055-836f-6808267ab03d","Type":"ContainerStarted","Data":"fe5869d7c53ede34b3a236480cc86beba82cb4a4489bebafb53ddee8e1badc0a"} Oct 30 15:10:06.548123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.548041 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:06.591629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.591580 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" podStartSLOduration=1.591566794 podStartE2EDuration="1.591566794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:06.588754766 +0000 UTC m=+1310.547796171" watchObservedRunningTime="2026-10-30 15:10:06.591566794 +0000 UTC m=+1310.550608200" Oct 30 15:10:06.631927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:06.631882 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" podStartSLOduration=1.631868915 podStartE2EDuration="1.631868915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:06.631538679 +0000 UTC m=+1310.590580089" watchObservedRunningTime="2026-10-30 15:10:06.631868915 +0000 UTC m=+1310.590910323" Oct 30 15:10:09.034200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:09.034164 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s"] Oct 30 15:10:09.047654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:09.047628 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6953b4e4-xvm6s"] Oct 30 15:10:09.202169 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:10:09.202136 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:10:10.156419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:10.156372 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d890d9c1-7385-474a-bcec-244f7ec89ea8" path="/var/lib/kubelet/pods/d890d9c1-7385-474a-bcec-244f7ec89ea8/volumes" Oct 30 15:10:10.497922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:10.497847 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:10:11.886704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.886667 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78"] Oct 30 15:10:11.916118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.915929 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:11.926595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.926558 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78"] Oct 30 15:10:11.948371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.947168 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgpdc\" (UniqueName: \"kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc\") pod \"payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:11.948371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.947301 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:11.973371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.955245 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9254b0\"" Oct 30 15:10:11.973371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:11.972295 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9254b0-dockercfg-r7s69\"" Oct 30 15:10:12.054374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:12.053679 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:12.054374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:12.054083 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgpdc\" (UniqueName: \"kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc\") pod \"payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:12.055570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:12.055539 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:12.087527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:12.087490 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgpdc\" (UniqueName: \"kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc\") pod \"payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:12.276578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:12.276491 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:12.614964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:12.614934 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78"] Oct 30 15:10:12.615955 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:10:12.615928 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfd7637_5771_49c9_be4d_c068ba5c1247.slice/crio-6b36cdabfdcadc00d6bcb3a9232a45c914c931d78ab3fc3afa6015b62795129d WatchSource:0}: Error finding container 6b36cdabfdcadc00d6bcb3a9232a45c914c931d78ab3fc3afa6015b62795129d: Status 404 returned error can't find the container with id 6b36cdabfdcadc00d6bcb3a9232a45c914c931d78ab3fc3afa6015b62795129d Oct 30 15:10:13.624049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:13.624014 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" event={"ID":"0cfd7637-5771-49c9-be4d-c068ba5c1247","Type":"ContainerStarted","Data":"3f3a8566dbf30e13d35d3af1de3f5982d91eef7bcbac8dfb59510bd1377527a9"} Oct 30 15:10:13.624473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:13.624079 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:13.624473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:13.624102 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" event={"ID":"0cfd7637-5771-49c9-be4d-c068ba5c1247","Type":"ContainerStarted","Data":"6b36cdabfdcadc00d6bcb3a9232a45c914c931d78ab3fc3afa6015b62795129d"} Oct 30 15:10:13.669012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:13.668964 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" podStartSLOduration=2.668952848 podStartE2EDuration="2.668952848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:13.667509875 +0000 UTC m=+1317.626551281" watchObservedRunningTime="2026-10-30 15:10:13.668952848 +0000 UTC m=+1317.627994253" Oct 30 15:10:16.688030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:16.687999 6554 scope.go:122] "RemoveContainer" containerID="5a2b7e4b884f1c2f51943a4348d7ffaec0a28cb7bb611d4188788e8fe9149305" Oct 30 15:10:16.698638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:16.698621 6554 scope.go:122] "RemoveContainer" containerID="c82599470e1b96ab511215d8360398d83607afa9315bc94b4594e4f8dd5bcbea" Oct 30 15:10:16.707634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:16.707612 6554 scope.go:122] "RemoveContainer" containerID="272d484fa551661a75cbf6078a368d98f941c4202d97da54c79c15840a32e1f8" Oct 30 15:10:16.716446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:16.716425 6554 scope.go:122] "RemoveContainer" containerID="ada98a45ba987edc837beb32af20c71d9d7098e7675e8c58d409409df49617ea" Oct 30 15:10:16.725505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:16.725485 6554 scope.go:122] "RemoveContainer" containerID="905fb5bae65774419a29b7cc7c2aa0e08134fac50374323f2b638ffbc6d5398c" Oct 30 15:10:17.563831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:17.563802 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:10:17.564335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:17.564309 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:10:18.965313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:18.965282 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7"] Oct 30 15:10:18.975323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:18.975293 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:18.995825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:18.995797 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7"] Oct 30 15:10:19.004953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.004919 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7acff45-openshift-default-dockercfg-qjdhh\"" Oct 30 15:10:19.067000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.066923 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.066956 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067043 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067096 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067227 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067274 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067412 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067447 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.067596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.067570 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbx4\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168499 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168544 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168587 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbx4\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168607 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168626 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168784 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168904 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.168998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.168923 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.169060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.169038 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.169123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.169107 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.169182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.169116 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.169245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.169207 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.169419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.169400 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.169984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.169962 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.170897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.170876 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.171197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.171179 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.192893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.192862 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.192993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.192977 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbx4\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.290522 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.290470 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:19.614403 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:10:19.614372 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-390224bf825b321fac1c8735529cbd1896056ba232d77502fbd1d1fad0253bd3 WatchSource:0}: Error finding container 390224bf825b321fac1c8735529cbd1896056ba232d77502fbd1d1fad0253bd3: Status 404 returned error can't find the container with id 390224bf825b321fac1c8735529cbd1896056ba232d77502fbd1d1fad0253bd3 Oct 30 15:10:19.614647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.614514 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7"] Oct 30 15:10:19.617340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.617306 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:10:19.617458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.617421 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:10:19.617524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.617479 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:10:19.682685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:19.682615 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" event={"ID":"69c54aae-4a98-45eb-be48-ff6a4c21d35f","Type":"ContainerStarted","Data":"390224bf825b321fac1c8735529cbd1896056ba232d77502fbd1d1fad0253bd3"} Oct 30 15:10:20.691882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:20.691840 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" event={"ID":"69c54aae-4a98-45eb-be48-ff6a4c21d35f","Type":"ContainerStarted","Data":"437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c"} Oct 30 15:10:20.780214 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:20.780141 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podStartSLOduration=2.780121304 podStartE2EDuration="2.780121304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:20.753539459 +0000 UTC m=+1324.712580864" watchObservedRunningTime="2026-10-30 15:10:20.780121304 +0000 UTC m=+1324.739162710" Oct 30 15:10:21.291336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:21.291306 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:21.292753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:21.292726 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 15:10:21.292867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:21.292775 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 15:10:22.291805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:22.291768 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 15:10:22.292448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:22.292414 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 15:10:23.291723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:23.291682 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 15:10:23.291894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:23.291786 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 15:10:24.302406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:24.301772 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 15:10:24.302406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:24.301841 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 15:10:24.634783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:24.634696 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:10:25.291670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:25.291639 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 30 15:10:25.291835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:25.291682 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 30 15:10:26.415042 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:26.414807 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:26.416038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:26.416011 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:26.420421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:26.420396 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:10:31.033648 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:31.033615 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk"] Oct 30 15:10:31.049537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:31.049509 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db4e1d9c-hsvzk"] Oct 30 15:10:32.164311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:32.164272 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c53d18e-a065-4428-a006-e7b90d104548" path="/var/lib/kubelet/pods/7c53d18e-a065-4428-a006-e7b90d104548/volumes" Oct 30 15:10:39.204216 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:10:39.204179 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:10:39.724371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.724328 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl"] Oct 30 15:10:39.729203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.729181 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.749611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.749579 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl"] Oct 30 15:10:39.759221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.759197 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-97ead4-serving-cert\"" Oct 30 15:10:39.820666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.820630 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.820851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.820780 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tl57\" (UniqueName: \"kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.820851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.820844 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.820948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.820928 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.922683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.922645 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.922818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.922726 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tl57\" (UniqueName: \"kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.922818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.922761 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.922818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.922782 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.923388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.923342 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.925315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.925294 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.925684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.925663 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:39.961140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:39.961091 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tl57\" (UniqueName: \"kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57\") pod \"maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:40.048505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:40.048421 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:40.378869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:40.378836 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl"] Oct 30 15:10:40.908728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:40.908694 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" event={"ID":"b5ea4576-a5ed-494c-a471-eea268e6548a","Type":"ContainerStarted","Data":"ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d"} Oct 30 15:10:40.908906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:40.908779 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" event={"ID":"b5ea4576-a5ed-494c-a471-eea268e6548a","Type":"ContainerStarted","Data":"409b57a1ecdda2b8fb222106e10adc71de49cf7fc4abf8cacaee7a8d8d469d93"} Oct 30 15:10:40.908906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:40.908811 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:40.956071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:40.956018 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" podStartSLOduration=1.583266783 podStartE2EDuration="1.95600537s" totalImagesPullingTime="372.738587ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:40.953806268 +0000 UTC m=+1344.912847676" watchObservedRunningTime="2026-10-30 15:10:40.95600537 +0000 UTC m=+1344.915046775" Oct 30 15:10:45.796901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:45.796864 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr"] Oct 30 15:10:45.831439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:45.831412 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497"] Oct 30 15:10:46.506113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.506081 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr"] Oct 30 15:10:46.537375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.535250 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr"] Oct 30 15:10:46.537375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.535430 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.558743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.557505 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9254b0-serving-cert\"" Oct 30 15:10:46.760216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.760099 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.760587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.760555 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.760877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.760861 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.761060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.761042 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj877\" (UniqueName: \"kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.862393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.862331 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.862811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.862477 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.862811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.862526 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.862811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.862572 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj877\" (UniqueName: \"kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.862811 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:10:46.862714 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-serving-cert: secret "maas-api-e2e-inf-a-9254b0-serving-cert" not found Oct 30 15:10:46.862811 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:10:46.862778 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls podName:e78c1b17-0ced-4359-a4a6-6f89a5ed098c nodeName:}" failed. No retries permitted until 2026-10-30 15:10:47.362763925 +0000 UTC m=+1351.321805328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls") pod "maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" (UID: "e78c1b17-0ced-4359-a4a6-6f89a5ed098c") : secret "maas-api-e2e-inf-a-9254b0-serving-cert" not found Oct 30 15:10:46.863073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.863053 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.865145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.865111 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.888365 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.888327 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj877\" (UniqueName: \"kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:46.925993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:46.925964 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:10:47.371270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:47.371234 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:47.374193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:47.374164 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\") pod \"maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:47.478265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:47.478233 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:47.804061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:47.804034 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr"] Oct 30 15:10:47.805085 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:10:47.805049 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78c1b17_0ced_4359_a4a6_6f89a5ed098c.slice/crio-3fdfff13c0b87a8fc753db6afe6dbb1ab21d7d53e28b5599f787093bb8006577 WatchSource:0}: Error finding container 3fdfff13c0b87a8fc753db6afe6dbb1ab21d7d53e28b5599f787093bb8006577: Status 404 returned error can't find the container with id 3fdfff13c0b87a8fc753db6afe6dbb1ab21d7d53e28b5599f787093bb8006577 Oct 30 15:10:47.967797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:47.967769 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" event={"ID":"e78c1b17-0ced-4359-a4a6-6f89a5ed098c","Type":"ContainerStarted","Data":"3fdfff13c0b87a8fc753db6afe6dbb1ab21d7d53e28b5599f787093bb8006577"} Oct 30 15:10:48.976878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:48.976840 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" event={"ID":"e78c1b17-0ced-4359-a4a6-6f89a5ed098c","Type":"ContainerStarted","Data":"3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414"} Oct 30 15:10:48.977288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:48.976911 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:49.028219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:49.028169 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" podStartSLOduration=2.6548959 podStartE2EDuration="3.028156279s" totalImagesPullingTime="373.260379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:10:49.023756719 +0000 UTC m=+1352.982798124" watchObservedRunningTime="2026-10-30 15:10:49.028156279 +0000 UTC m=+1352.987197684" Oct 30 15:10:50.802773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:50.802527 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerName="payload-processing" containerID="cri-o://f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02" gracePeriod=25 Oct 30 15:10:50.855465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:50.855393 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerName="payload-pre-processing" containerID="cri-o://0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f" gracePeriod=25 Oct 30 15:10:54.990327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:54.990300 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:10:58.253884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:58.253794 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:10:58.254284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:10:58.253951 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:11:06.384262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.384201 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb"] Oct 30 15:11:06.389430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.389407 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.405870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.405850 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-76473c-openshift-default-dockercfg-p7vs4\"" Oct 30 15:11:06.424446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.424422 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb"] Oct 30 15:11:06.523512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.523484 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.523665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.523532 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.523732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.523693 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznqr\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.523854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.523831 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.523934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.523919 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.524056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.524038 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.524105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.524082 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.524163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.524121 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.524220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.524173 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.625938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.625901 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.625938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.625951 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.625994 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626033 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626105 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznqr\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626201 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626272 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626342 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626591 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.626725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.626700 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.627065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.627034 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.627168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.627068 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.627305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.627282 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.627475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.627455 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.631027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.631003 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.632825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.632803 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.665904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.665873 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.667257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.667221 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznqr\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr\") pod \"e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:06.704073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:06.704037 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:07.046187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.046161 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb"] Oct 30 15:11:07.047972 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:11:07.047947 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c91760f_f50c_4ce7_800d_a8c40cee1ef0.slice/crio-32dc8057bc04d97f5bd86326a154d46f7838fbd741ad7af53b6b8b803eae1b3c WatchSource:0}: Error finding container 32dc8057bc04d97f5bd86326a154d46f7838fbd741ad7af53b6b8b803eae1b3c: Status 404 returned error can't find the container with id 32dc8057bc04d97f5bd86326a154d46f7838fbd741ad7af53b6b8b803eae1b3c Oct 30 15:11:07.050940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.050907 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:11:07.051033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.050974 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:11:07.051033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.051006 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:11:07.307980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.307885 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" event={"ID":"7c91760f-f50c-4ce7-800d-a8c40cee1ef0","Type":"ContainerStarted","Data":"3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3"} Oct 30 15:11:07.307980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.307958 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" event={"ID":"7c91760f-f50c-4ce7-800d-a8c40cee1ef0","Type":"ContainerStarted","Data":"32dc8057bc04d97f5bd86326a154d46f7838fbd741ad7af53b6b8b803eae1b3c"} Oct 30 15:11:07.358770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.358723 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podStartSLOduration=1.358708309 podStartE2EDuration="1.358708309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:07.355333162 +0000 UTC m=+1371.314374568" watchObservedRunningTime="2026-10-30 15:11:07.358708309 +0000 UTC m=+1371.317749713" Oct 30 15:11:07.704920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:07.704886 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:08.278323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:08.278282 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:11:08.278968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:08.278914 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:11:08.705905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:08.705876 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 15:11:08.706418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:08.705935 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 15:11:09.206593 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:09.206555 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:11:09.705065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:09.705028 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 15:11:09.705240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:09.705098 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 15:11:10.447573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:10.447542 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg"] Oct 30 15:11:10.447977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:10.447761 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" containerID="cri-o://52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51" gracePeriod=30 Oct 30 15:11:10.705070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:10.704986 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 15:11:10.705070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:10.705044 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 15:11:11.141378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.141334 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:11:11.233489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.233457 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:11:11.234279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.234247 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql\") pod \"3fee4b6a-11e1-46e6-8636-4626b57c203c\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " Oct 30 15:11:11.234514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.234491 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume\") pod \"3fee4b6a-11e1-46e6-8636-4626b57c203c\" (UID: \"3fee4b6a-11e1-46e6-8636-4626b57c203c\") " Oct 30 15:11:11.236707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.236674 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume" pod "3fee4b6a-11e1-46e6-8636-4626b57c203c" (UID: "3fee4b6a-11e1-46e6-8636-4626b57c203c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:11.238376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.238338 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql" pod "3fee4b6a-11e1-46e6-8636-4626b57c203c" (UID: "3fee4b6a-11e1-46e6-8636-4626b57c203c"). InnerVolumeSpecName "kube-api-access-7b6ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:11.337464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.337374 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd\" (UniqueName: \"kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd\") pod \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " Oct 30 15:11:11.337637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.337582 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume\") pod \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\" (UID: \"69b10318-ff5f-41cf-bcd2-ed79fdfd526a\") " Oct 30 15:11:11.337950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.337926 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume" pod "69b10318-ff5f-41cf-bcd2-ed79fdfd526a" (UID: "69b10318-ff5f-41cf-bcd2-ed79fdfd526a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:11.338179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.338158 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b6ql\" (UniqueName: \"kubernetes.io/projected/3fee4b6a-11e1-46e6-8636-4626b57c203c-kube-api-access-7b6ql\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:11.338256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.338185 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:11.338256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.338201 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fee4b6a-11e1-46e6-8636-4626b57c203c-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:11.339603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.339579 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd" pod "69b10318-ff5f-41cf-bcd2-ed79fdfd526a" (UID: "69b10318-ff5f-41cf-bcd2-ed79fdfd526a"). InnerVolumeSpecName "kube-api-access-gr9jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:11.349929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.349902 6554 generic.go:356] "Generic (PLEG): container finished" podID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerID="f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02" exitCode=0 Oct 30 15:11:11.350043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.349969 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" event={"ID":"3fee4b6a-11e1-46e6-8636-4626b57c203c","Type":"ContainerDied","Data":"f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02"} Oct 30 15:11:11.350043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.350017 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" event={"ID":"3fee4b6a-11e1-46e6-8636-4626b57c203c","Type":"ContainerDied","Data":"93902765405c57c876e0dcbdd1b4c28ef3880257f3aecc02952fb545e1146e48"} Oct 30 15:11:11.350167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.350043 6554 scope.go:122] "RemoveContainer" containerID="f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02" Oct 30 15:11:11.350487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.350468 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr" Oct 30 15:11:11.351741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.351717 6554 generic.go:356] "Generic (PLEG): container finished" podID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerID="0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f" exitCode=0 Oct 30 15:11:11.351853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.351778 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" Oct 30 15:11:11.351853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.351789 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" event={"ID":"69b10318-ff5f-41cf-bcd2-ed79fdfd526a","Type":"ContainerDied","Data":"0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f"} Oct 30 15:11:11.351853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.351826 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497" event={"ID":"69b10318-ff5f-41cf-bcd2-ed79fdfd526a","Type":"ContainerDied","Data":"4454b645766019531a4a569b21df1bd9ac0cb7cc2b1284fce562dcf4b7607344"} Oct 30 15:11:11.389588 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.389570 6554 scope.go:122] "RemoveContainer" containerID="f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02" Oct 30 15:11:11.390235 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:11.389908 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02\": container with ID starting with f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02 not found: ID does not exist" containerID="f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02" Oct 30 15:11:11.390235 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.389934 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02"} err="failed to get container status \"f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02\": rpc error: code = NotFound desc = could not find container \"f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02\": container with ID starting with f303b92a3da42c4bddd8f5bb7b12623cad54362501be7a40b8b8bb841bbf3f02 not found: ID does not exist" Oct 30 15:11:11.390235 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.389952 6554 scope.go:122] "RemoveContainer" containerID="0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f" Oct 30 15:11:11.401266 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.401251 6554 scope.go:122] "RemoveContainer" containerID="0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f" Oct 30 15:11:11.401519 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:11.401502 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f\": container with ID starting with 0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f not found: ID does not exist" containerID="0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f" Oct 30 15:11:11.401577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.401526 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f"} err="failed to get container status \"0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f\": rpc error: code = NotFound desc = could not find container \"0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f\": container with ID starting with 0e2c663ba9f67dfb9df2b3503cb5cd26e50fe88f22e0b5e763994170ff0e2d9f not found: ID does not exist" Oct 30 15:11:11.437223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.437177 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497"] Oct 30 15:11:11.440008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.439980 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr9jd\" (UniqueName: \"kubernetes.io/projected/69b10318-ff5f-41cf-bcd2-ed79fdfd526a-kube-api-access-gr9jd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:11.441088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.441068 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-m5497"] Oct 30 15:11:11.484478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.484448 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr"] Oct 30 15:11:11.488131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.488109 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-44mqr"] Oct 30 15:11:11.704817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.704787 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 15:11:11.704970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:11.704850 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 15:11:12.183578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:12.183551 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" path="/var/lib/kubelet/pods/3fee4b6a-11e1-46e6-8636-4626b57c203c/volumes" Oct 30 15:11:12.184247 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:12.184229 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" path="/var/lib/kubelet/pods/69b10318-ff5f-41cf-bcd2-ed79fdfd526a/volumes" Oct 30 15:11:12.704634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:12.704603 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 15:11:12.705019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:12.704653 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 15:11:14.054272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:14.054239 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:14.054970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:14.054942 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:14.056319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:14.056298 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:11:15.731657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.731591 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863323 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863392 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863535 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863614 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863665 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863745 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863772 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863920 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.863957 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data\") pod \"3f83f289-4683-4f0a-a85a-41045ade9be9\" (UID: \"3f83f289-4683-4f0a-a85a-41045ade9be9\") " Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.903886 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.904063 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.904185 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.904415 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:15.910744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.904541 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.920669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.920601 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:15.933732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.933299 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "kube-api-access-tf65s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:15.943406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.935830 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968826 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968857 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf65s\" (UniqueName: \"kubernetes.io/projected/3f83f289-4683-4f0a-a85a-41045ade9be9-kube-api-access-tf65s\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968871 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f83f289-4683-4f0a-a85a-41045ade9be9-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968884 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968897 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968911 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968925 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.976506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.968938 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:15.977278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:15.977242 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy" pod "3f83f289-4683-4f0a-a85a-41045ade9be9" (UID: "3f83f289-4683-4f0a-a85a-41045ade9be9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:16.070470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.070384 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f83f289-4683-4f0a-a85a-41045ade9be9-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:16.417749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.417711 6554 generic.go:356] "Generic (PLEG): container finished" podID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerID="52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51" exitCode=0 Oct 30 15:11:16.417953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.417783 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" Oct 30 15:11:16.417953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.417797 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" event={"ID":"3f83f289-4683-4f0a-a85a-41045ade9be9","Type":"ContainerDied","Data":"52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51"} Oct 30 15:11:16.417953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.417867 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg" event={"ID":"3f83f289-4683-4f0a-a85a-41045ade9be9","Type":"ContainerDied","Data":"7ec10a1c1dff4b91da49db48df71835d7fdc23aebc73f3f6f60a28cc65f8df5d"} Oct 30 15:11:16.417953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.417890 6554 scope.go:122] "RemoveContainer" containerID="52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51" Oct 30 15:11:16.429322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.429305 6554 scope.go:122] "RemoveContainer" containerID="52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51" Oct 30 15:11:16.430072 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:16.429661 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51\": container with ID starting with 52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51 not found: ID does not exist" containerID="52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51" Oct 30 15:11:16.430072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.429702 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51"} err="failed to get container status \"52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51\": rpc error: code = NotFound desc = could not find container \"52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51\": container with ID starting with 52514f32bd0a20c683814515412fc2885523eb3ae9853351efd5474fe27b5b51 not found: ID does not exist" Oct 30 15:11:16.495234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.495210 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg"] Oct 30 15:11:16.498781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.498761 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-gsdvg"] Oct 30 15:11:16.774680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.774594 6554 scope.go:122] "RemoveContainer" containerID="02ce499991908b18696c27183e06bd129c155b250798c5dcfa155a8ae0f0f616" Oct 30 15:11:16.785794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.785775 6554 scope.go:122] "RemoveContainer" containerID="e7bf4b82cfb77148a62795d05badc1fac0dadaf10a013c723e56db55ddbdd4cf" Oct 30 15:11:16.795624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:16.795603 6554 scope.go:122] "RemoveContainer" containerID="a4f2f934f5eb83b5f0be1c9f2f6877b0728c90c62655af3f0e84d95936a2f7d1" Oct 30 15:11:18.166086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:18.166049 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" path="/var/lib/kubelet/pods/3f83f289-4683-4f0a-a85a-41045ade9be9/volumes" Oct 30 15:11:25.719645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.719612 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz"] Oct 30 15:11:25.720105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.719794 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" containerID="cri-o://6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a" gracePeriod=30 Oct 30 15:11:25.733123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.733093 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh"] Oct 30 15:11:25.733331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.733306 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" containerID="cri-o://6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4" gracePeriod=30 Oct 30 15:11:25.737097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.737072 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t"] Oct 30 15:11:25.737239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.737223 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" containerID="cri-o://2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09" gracePeriod=30 Oct 30 15:11:25.745959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.745934 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt"] Oct 30 15:11:25.746141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:25.746120 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" containerID="cri-o://f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e" gracePeriod=30 Oct 30 15:11:26.290269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.290231 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl"] Oct 30 15:11:26.291492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291461 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" Oct 30 15:11:26.291492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291490 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" Oct 30 15:11:26.291688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291541 6554 cpu_manager.go:414] "containerMap: removing container" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerName="payload-pre-processing" Oct 30 15:11:26.291688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291550 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerName="payload-pre-processing" Oct 30 15:11:26.291688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291595 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerName="payload-processing" Oct 30 15:11:26.291688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291602 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerName="payload-processing" Oct 30 15:11:26.291902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291780 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fee4b6a-11e1-46e6-8636-4626b57c203c" containerName="payload-processing" Oct 30 15:11:26.291902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291802 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f83f289-4683-4f0a-a85a-41045ade9be9" containerName="istio-proxy" Oct 30 15:11:26.291902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.291820 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b10318-ff5f-41cf-bcd2-ed79fdfd526a" containerName="payload-pre-processing" Oct 30 15:11:26.298117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.298092 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.313653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.313624 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl"] Oct 30 15:11:26.387461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.385180 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-147bdb\"" Oct 30 15:11:26.387461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.385262 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-147bdb-dockercfg-lbrmp\"" Oct 30 15:11:26.482850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.482102 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjxbn\" (UniqueName: \"kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn\") pod \"payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.503397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.503341 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.533781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.533744 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d"] Oct 30 15:11:26.534301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.534264 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" containerID="cri-o://dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd" gracePeriod=30 Oct 30 15:11:26.544478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.542450 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc"] Oct 30 15:11:26.544478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.542683 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" containerID="cri-o://328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5" gracePeriod=30 Oct 30 15:11:26.547598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.547577 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2"] Oct 30 15:11:26.547894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.547870 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" containerID="cri-o://8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2" gracePeriod=30 Oct 30 15:11:26.566414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.566381 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j"] Oct 30 15:11:26.592202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.592160 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" containerID="cri-o://e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1" gracePeriod=30 Oct 30 15:11:26.630411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.629737 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.630411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.629847 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjxbn\" (UniqueName: \"kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn\") pod \"payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.631193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.631168 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.655546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.655514 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjxbn\" (UniqueName: \"kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn\") pod \"payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:26.917014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:26.916986 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:27.228906 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:11:27.228878 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1aa13d_72fd_4e10_9733_4773fc7c69b5.slice/crio-0b8eb7a31aad4290b561f5591d62851fe82771bebc1a935924f7e2eed465b765 WatchSource:0}: Error finding container 0b8eb7a31aad4290b561f5591d62851fe82771bebc1a935924f7e2eed465b765: Status 404 returned error can't find the container with id 0b8eb7a31aad4290b561f5591d62851fe82771bebc1a935924f7e2eed465b765 Oct 30 15:11:27.229105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:27.229084 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl"] Oct 30 15:11:27.531479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:27.531401 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" event={"ID":"6c1aa13d-72fd-4e10-9733-4773fc7c69b5","Type":"ContainerStarted","Data":"be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f"} Oct 30 15:11:27.531479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:27.531476 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" event={"ID":"6c1aa13d-72fd-4e10-9733-4773fc7c69b5","Type":"ContainerStarted","Data":"0b8eb7a31aad4290b561f5591d62851fe82771bebc1a935924f7e2eed465b765"} Oct 30 15:11:27.531647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:27.531499 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:27.577204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:27.577159 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" podStartSLOduration=1.577147096 podStartE2EDuration="1.577147096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:27.574666927 +0000 UTC m=+1391.533708333" watchObservedRunningTime="2026-10-30 15:11:27.577147096 +0000 UTC m=+1391.536188501" Oct 30 15:11:31.102402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.102369 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:11:31.155586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.155561 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:11:31.159529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.159505 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:11:31.165153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.165103 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:11:31.228342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228303 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.228773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228748 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.228880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228794 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.228880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228838 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.228984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228887 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.228984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228917 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.228984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228945 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.229141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.228985 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.229141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.229018 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.229141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.229085 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.229960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.229935 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.230117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.229998 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.230117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230047 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.230117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230074 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230122 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230154 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230209 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230242 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230285 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230326 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230448 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230502 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230598 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230635 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230740 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230784 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230830 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230878 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.230921 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231007 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231080 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231139 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231177 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket\") pod \"4767fe86-4975-44bb-a86f-bafa6b10f05a\" (UID: \"4767fe86-4975-44bb-a86f-bafa6b10f05a\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231256 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data\") pod \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\" (UID: \"f7a2cff1-a2e4-4c9d-a474-279529d4a336\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231291 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert\") pod \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\" (UID: \"3fc78931-2fc6-42b5-94a1-8a2d98513cf6\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.231331 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5\") pod \"154c6b17-aa49-46c2-9cda-7d795d69492f\" (UID: \"154c6b17-aa49-46c2-9cda-7d795d69492f\") " Oct 30 15:11:31.234513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.234164 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:31.237041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.237013 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.239605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.239566 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.240983 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.240958 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.241133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.241110 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:31.241531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.241503 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:31.242054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.242030 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.242527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.242416 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.242713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.242644 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:31.242820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.242686 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.242820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.242740 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.243036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.242917 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.243866 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.243954 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.244257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244068 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244085 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244115 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:31.244586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244329 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244573 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244583 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244771 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.244884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244820 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.245012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.244989 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.245606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.245579 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "kube-api-access-jhm9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.246666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.246646 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:31.246869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.246844 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "kube-api-access-psndj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.246962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.246941 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.247036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.246997 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "kube-api-access-pvtmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.248083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.248034 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:31.248221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.248202 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy" pod "f7a2cff1-a2e4-4c9d-a474-279529d4a336" (UID: "f7a2cff1-a2e4-4c9d-a474-279529d4a336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.248692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.248616 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy" pod "3fc78931-2fc6-42b5-94a1-8a2d98513cf6" (UID: "3fc78931-2fc6-42b5-94a1-8a2d98513cf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.248692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.248664 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "kube-api-access-vhhl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.248850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.248694 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.248850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.248726 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:31.250724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.250694 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo" pod "154c6b17-aa49-46c2-9cda-7d795d69492f" (UID: "154c6b17-aa49-46c2-9cda-7d795d69492f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:31.250808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.250751 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy" pod "4767fe86-4975-44bb-a86f-bafa6b10f05a" (UID: "4767fe86-4975-44bb-a86f-bafa6b10f05a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:31.334186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334154 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334183 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334193 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334202 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334211 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4767fe86-4975-44bb-a86f-bafa6b10f05a-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334220 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334227 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334236 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334244 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334252 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334260 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334268 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334276 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334285 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334294 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334303 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334311 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334319 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334327 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334335 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhm9j\" (UniqueName: \"kubernetes.io/projected/4767fe86-4975-44bb-a86f-bafa6b10f05a-kube-api-access-jhm9j\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334342 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334366 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334374 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334382 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhhl5\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-kube-api-access-vhhl5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334391 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334399 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4767fe86-4975-44bb-a86f-bafa6b10f05a-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334407 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvtmz\" (UniqueName: \"kubernetes.io/projected/f7a2cff1-a2e4-4c9d-a474-279529d4a336-kube-api-access-pvtmz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334415 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334423 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154c6b17-aa49-46c2-9cda-7d795d69492f-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334430 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334439 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334447 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334456 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7a2cff1-a2e4-4c9d-a474-279529d4a336-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334464 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334473 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154c6b17-aa49-46c2-9cda-7d795d69492f-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.334518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.334481 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psndj\" (UniqueName: \"kubernetes.io/projected/3fc78931-2fc6-42b5-94a1-8a2d98513cf6-kube-api-access-psndj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:31.580673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.580638 6554 generic.go:356] "Generic (PLEG): container finished" podID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerID="6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4" exitCode=0 Oct 30 15:11:31.580842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.580801 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" event={"ID":"f7a2cff1-a2e4-4c9d-a474-279529d4a336","Type":"ContainerDied","Data":"6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4"} Oct 30 15:11:31.580908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.580883 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" event={"ID":"f7a2cff1-a2e4-4c9d-a474-279529d4a336","Type":"ContainerDied","Data":"24f169781c4f779db782fced3036a7c4a40c2f7beb7a916588f833d48b599c22"} Oct 30 15:11:31.580960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.580930 6554 scope.go:122] "RemoveContainer" containerID="6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4" Oct 30 15:11:31.581082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.580767 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh" Oct 30 15:11:31.584833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.584813 6554 generic.go:356] "Generic (PLEG): container finished" podID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerID="2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09" exitCode=0 Oct 30 15:11:31.584929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.584916 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" event={"ID":"3fc78931-2fc6-42b5-94a1-8a2d98513cf6","Type":"ContainerDied","Data":"2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09"} Oct 30 15:11:31.584968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.584952 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" event={"ID":"3fc78931-2fc6-42b5-94a1-8a2d98513cf6","Type":"ContainerDied","Data":"44338e28782d4efe2117eb0b742734b14af4308578635b2a96a02d9843614712"} Oct 30 15:11:31.585013 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.585004 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t" Oct 30 15:11:31.586846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.586821 6554 generic.go:356] "Generic (PLEG): container finished" podID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerID="f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e" exitCode=0 Oct 30 15:11:31.586955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.586884 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" Oct 30 15:11:31.586955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.586920 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" event={"ID":"154c6b17-aa49-46c2-9cda-7d795d69492f","Type":"ContainerDied","Data":"f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e"} Oct 30 15:11:31.587082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.586974 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" event={"ID":"154c6b17-aa49-46c2-9cda-7d795d69492f","Type":"ContainerDied","Data":"5ebc7f533b0ccb8d482207c34622e51bfca5d66320c3954594aeed17cd36abdf"} Oct 30 15:11:31.588430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.588409 6554 generic.go:356] "Generic (PLEG): container finished" podID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerID="6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a" exitCode=0 Oct 30 15:11:31.588561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.588549 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" event={"ID":"4767fe86-4975-44bb-a86f-bafa6b10f05a","Type":"ContainerDied","Data":"6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a"} Oct 30 15:11:31.588625 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.588581 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" event={"ID":"4767fe86-4975-44bb-a86f-bafa6b10f05a","Type":"ContainerDied","Data":"66c17d72f1b951d22b0317c9dcbf2a8da1a56c1ad1098d23cd4b348f6d1dfd53"} Oct 30 15:11:31.588678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.588630 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz" Oct 30 15:11:31.831182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.830830 6554 scope.go:122] "RemoveContainer" containerID="6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4" Oct 30 15:11:31.831780 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:31.831408 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4\": container with ID starting with 6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4 not found: ID does not exist" containerID="6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4" Oct 30 15:11:31.831780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.831450 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4"} err="failed to get container status \"6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4\": rpc error: code = NotFound desc = could not find container \"6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4\": container with ID starting with 6482797c13d79633e14c4cbcf399bdd210f2664f9656d0ed8a4e37926f680fa4 not found: ID does not exist" Oct 30 15:11:31.831780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.831477 6554 scope.go:122] "RemoveContainer" containerID="2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09" Oct 30 15:11:31.857252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.856860 6554 scope.go:122] "RemoveContainer" containerID="2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09" Oct 30 15:11:31.857415 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:31.857225 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09\": container with ID starting with 2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09 not found: ID does not exist" containerID="2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09" Oct 30 15:11:31.857415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.857327 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09"} err="failed to get container status \"2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09\": rpc error: code = NotFound desc = could not find container \"2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09\": container with ID starting with 2e918d893b0aa9c3ae0baeb94b112640b9fd34d2586488c6ab2c9fb57ca12d09 not found: ID does not exist" Oct 30 15:11:31.857415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.857370 6554 scope.go:122] "RemoveContainer" containerID="f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e" Oct 30 15:11:31.869627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.869278 6554 scope.go:122] "RemoveContainer" containerID="f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e" Oct 30 15:11:31.869627 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:31.869558 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e\": container with ID starting with f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e not found: ID does not exist" containerID="f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e" Oct 30 15:11:31.869627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.869600 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e"} err="failed to get container status \"f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e\": rpc error: code = NotFound desc = could not find container \"f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e\": container with ID starting with f38bcbfa0a145568ed14c9c48062456dbfa648b9c225a800a2cbb3dd056a8d0e not found: ID does not exist" Oct 30 15:11:31.869762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.869632 6554 scope.go:122] "RemoveContainer" containerID="6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a" Oct 30 15:11:31.880644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.880557 6554 scope.go:122] "RemoveContainer" containerID="6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a" Oct 30 15:11:31.882206 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:31.882048 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a\": container with ID starting with 6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a not found: ID does not exist" containerID="6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a" Oct 30 15:11:31.882206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.882083 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a"} err="failed to get container status \"6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a\": rpc error: code = NotFound desc = could not find container \"6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a\": container with ID starting with 6a413bee0fcc0c7e92fb7890ac12705578446758eaefb6569ea439cebc0c6b9a not found: ID does not exist" Oct 30 15:11:31.888582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.888560 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz"] Oct 30 15:11:31.892690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.892672 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-bc2jz"] Oct 30 15:11:31.958859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.949805 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t"] Oct 30 15:11:31.958859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.955939 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-97v9t"] Oct 30 15:11:31.986413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.981010 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh"] Oct 30 15:11:32.021629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:31.987332 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-dj6qh"] Oct 30 15:11:32.021629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.008091 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:11:32.045660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.045629 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt"] Oct 30 15:11:32.048015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.047939 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt"] Oct 30 15:11:32.072693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.072538 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:11:32.074638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.074613 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:11:32.077047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.077031 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:11:32.122429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.122339 6554 patch_prober.go:29] interesting pod/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:11:32.122429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.122406 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-2c8d89-openshift-default-6569c77797-vlpmt" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:11:32.160937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.160901 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" path="/var/lib/kubelet/pods/154c6b17-aa49-46c2-9cda-7d795d69492f/volumes" Oct 30 15:11:32.161797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.161773 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" path="/var/lib/kubelet/pods/3fc78931-2fc6-42b5-94a1-8a2d98513cf6/volumes" Oct 30 15:11:32.162531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.162509 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" path="/var/lib/kubelet/pods/4767fe86-4975-44bb-a86f-bafa6b10f05a/volumes" Oct 30 15:11:32.163698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.163676 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" path="/var/lib/kubelet/pods/f7a2cff1-a2e4-4c9d-a474-279529d4a336/volumes" Oct 30 15:11:32.172817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.172795 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.172916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.172840 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.172916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.172887 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.173177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173153 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.173294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173157 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.173294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173217 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.173294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173261 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.173294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173291 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.173576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173338 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.173576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173387 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.173576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173423 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.173576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173478 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.173576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173489 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.173576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173520 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173711 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173724 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173749 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173806 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173848 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173922 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.173952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173954 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.173991 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174020 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174070 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174125 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174133 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174158 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174197 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174233 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.174309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174307 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174334 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174368 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174382 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174415 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174442 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174490 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket\") pod \"52292a7b-4a45-4110-af04-a35cbb09893f\" (UID: \"52292a7b-4a45-4110-af04-a35cbb09893f\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174522 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174575 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174614 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174658 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket\") pod \"21fe275d-7a84-478c-863d-2db9076587f4\" (UID: \"21fe275d-7a84-478c-863d-2db9076587f4\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174690 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data\") pod \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\" (UID: \"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6\") " Oct 30 15:11:32.174793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.174719 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token\") pod \"9e5a0723-eb84-458e-a15e-180c90d45e37\" (UID: \"9e5a0723-eb84-458e-a15e-180c90d45e37\") " Oct 30 15:11:32.175706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175681 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.175857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175839 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.175968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175955 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52292a7b-4a45-4110-af04-a35cbb09893f-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.176063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176050 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.176158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176145 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.176243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176231 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21fe275d-7a84-478c-863d-2db9076587f4-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.176317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175680 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.176415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175713 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.176511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175865 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.176587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175939 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.176662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.175955 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.176736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176148 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.176815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176309 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.176893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176625 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:32.176974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176818 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.177047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.176853 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.177339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.177319 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.177769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.177747 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:32.178968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.178729 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.178968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.178737 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.179114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.178978 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "kube-api-access-zf246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.179435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.179233 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.179952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.179669 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.179952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.179704 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:32.179952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.179738 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "kube-api-access-vgjqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.180133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.179985 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.180203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.180176 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.180203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.180187 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:32.181090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.181064 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.181562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.181541 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "kube-api-access-db75v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.181651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.181640 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.182235 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.182207 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49" pod "52292a7b-4a45-4110-af04-a35cbb09893f" (UID: "52292a7b-4a45-4110-af04-a35cbb09893f"). InnerVolumeSpecName "kube-api-access-spf49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.182662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.182637 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo" pod "21fe275d-7a84-478c-863d-2db9076587f4" (UID: "21fe275d-7a84-478c-863d-2db9076587f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:32.183065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.183041 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:11:32.183580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.183554 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token" pod "9e5a0723-eb84-458e-a15e-180c90d45e37" (UID: "9e5a0723-eb84-458e-a15e-180c90d45e37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:32.184117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.184098 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo" pod "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" (UID: "9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:11:32.279044 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279010 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279046 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db75v\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-kube-api-access-db75v\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279079 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279094 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279109 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spf49\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-kube-api-access-spf49\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279122 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279152 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279167 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279181 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52292a7b-4a45-4110-af04-a35cbb09893f-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279194 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279208 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgjqd\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-kube-api-access-vgjqd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279565 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279582 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279597 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279610 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279624 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52292a7b-4a45-4110-af04-a35cbb09893f-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279636 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279649 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279662 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e5a0723-eb84-458e-a15e-180c90d45e37-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279675 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279687 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21fe275d-7a84-478c-863d-2db9076587f4-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279700 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21fe275d-7a84-478c-863d-2db9076587f4-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279713 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279727 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279740 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279752 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52292a7b-4a45-4110-af04-a35cbb09893f-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279764 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e5a0723-eb84-458e-a15e-180c90d45e37-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279776 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279792 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zf246\" (UniqueName: \"kubernetes.io/projected/21fe275d-7a84-478c-863d-2db9076587f4-kube-api-access-zf246\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.279848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.279806 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:32.597703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.597667 6554 generic.go:356] "Generic (PLEG): container finished" podID="52292a7b-4a45-4110-af04-a35cbb09893f" containerID="e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1" exitCode=0 Oct 30 15:11:32.597910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.597712 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" Oct 30 15:11:32.597910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.597707 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" event={"ID":"52292a7b-4a45-4110-af04-a35cbb09893f","Type":"ContainerDied","Data":"e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1"} Oct 30 15:11:32.597910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.597781 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j" event={"ID":"52292a7b-4a45-4110-af04-a35cbb09893f","Type":"ContainerDied","Data":"42ddd2993f0a35e1c5f16e63236e5a3021f4416eea09a4b6304e717dba60b331"} Oct 30 15:11:32.597910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.597811 6554 scope.go:122] "RemoveContainer" containerID="e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1" Oct 30 15:11:32.599480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.599451 6554 generic.go:356] "Generic (PLEG): container finished" podID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerID="328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5" exitCode=0 Oct 30 15:11:32.599596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.599498 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" Oct 30 15:11:32.599596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.599512 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" event={"ID":"9e5a0723-eb84-458e-a15e-180c90d45e37","Type":"ContainerDied","Data":"328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5"} Oct 30 15:11:32.599596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.599558 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc" event={"ID":"9e5a0723-eb84-458e-a15e-180c90d45e37","Type":"ContainerDied","Data":"8b2c176dc2d241c3c7a4b7c5dae7f90c96d43cb36e81f78b5a477b65355f9b26"} Oct 30 15:11:32.602156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.602131 6554 generic.go:356] "Generic (PLEG): container finished" podID="21fe275d-7a84-478c-863d-2db9076587f4" containerID="8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2" exitCode=0 Oct 30 15:11:32.602265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.602183 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" Oct 30 15:11:32.602265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.602211 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" event={"ID":"21fe275d-7a84-478c-863d-2db9076587f4","Type":"ContainerDied","Data":"8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2"} Oct 30 15:11:32.602265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.602253 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2" event={"ID":"21fe275d-7a84-478c-863d-2db9076587f4","Type":"ContainerDied","Data":"1994ac9572432a3ee3f4a85ad41da51dfcdcc809e24e227550134a633951293c"} Oct 30 15:11:32.603896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.603867 6554 generic.go:356] "Generic (PLEG): container finished" podID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerID="dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd" exitCode=0 Oct 30 15:11:32.604016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.603938 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" Oct 30 15:11:32.604016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.603952 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" event={"ID":"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6","Type":"ContainerDied","Data":"dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd"} Oct 30 15:11:32.604016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.603998 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d" event={"ID":"9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6","Type":"ContainerDied","Data":"8acd3a4ca32f00f6ea8d925cb5b11de45b41eda855280d744360ef4ebb6285a4"} Oct 30 15:11:32.610831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.610814 6554 scope.go:122] "RemoveContainer" containerID="e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1" Oct 30 15:11:32.611088 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:32.611070 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1\": container with ID starting with e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1 not found: ID does not exist" containerID="e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1" Oct 30 15:11:32.611151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.611094 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1"} err="failed to get container status \"e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1\": rpc error: code = NotFound desc = could not find container \"e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1\": container with ID starting with e6fa6ab256ba789461141e337e9a0cf778d477fea7b22fe3a88c9c8a0a9d73f1 not found: ID does not exist" Oct 30 15:11:32.611151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.611117 6554 scope.go:122] "RemoveContainer" containerID="328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5" Oct 30 15:11:32.621763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.621748 6554 scope.go:122] "RemoveContainer" containerID="328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5" Oct 30 15:11:32.622051 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:32.622033 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5\": container with ID starting with 328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5 not found: ID does not exist" containerID="328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5" Oct 30 15:11:32.622133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.622062 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5"} err="failed to get container status \"328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5\": rpc error: code = NotFound desc = could not find container \"328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5\": container with ID starting with 328c068b21f783cb10ab0ecd7c95d66d95cda290c88613686753ce45667c49f5 not found: ID does not exist" Oct 30 15:11:32.622133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.622084 6554 scope.go:122] "RemoveContainer" containerID="8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2" Oct 30 15:11:32.636035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.636008 6554 scope.go:122] "RemoveContainer" containerID="8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2" Oct 30 15:11:32.636624 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:32.636594 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2\": container with ID starting with 8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2 not found: ID does not exist" containerID="8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2" Oct 30 15:11:32.636751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.636633 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2"} err="failed to get container status \"8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2\": rpc error: code = NotFound desc = could not find container \"8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2\": container with ID starting with 8508ce26477b8c15ae3f5059f32894ee2962e90e345fd29bc0ab13bc07ce4de2 not found: ID does not exist" Oct 30 15:11:32.636751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.636682 6554 scope.go:122] "RemoveContainer" containerID="dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd" Oct 30 15:11:32.652569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.652531 6554 scope.go:122] "RemoveContainer" containerID="dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd" Oct 30 15:11:32.652934 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:32.652903 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd\": container with ID starting with dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd not found: ID does not exist" containerID="dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd" Oct 30 15:11:32.653071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.652943 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd"} err="failed to get container status \"dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd\": rpc error: code = NotFound desc = could not find container \"dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd\": container with ID starting with dfba4c49f369864aca1a07fc490e5b07c17f5c3986a4cd1e8dad8e7d884c29cd not found: ID does not exist" Oct 30 15:11:32.706374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.704203 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j"] Oct 30 15:11:32.719765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.719734 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-7629j"] Oct 30 15:11:32.811730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.811411 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc"] Oct 30 15:11:32.818891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.818865 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-9fqxc"] Oct 30 15:11:32.856136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.856073 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d"] Oct 30 15:11:32.872671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.872645 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-qdz4d"] Oct 30 15:11:32.919112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.919082 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2"] Oct 30 15:11:32.924924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:32.924902 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-83ff4e-openshift-default-7fb46f8d47-5mdz2"] Oct 30 15:11:34.156843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:34.156811 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21fe275d-7a84-478c-863d-2db9076587f4" path="/var/lib/kubelet/pods/21fe275d-7a84-478c-863d-2db9076587f4/volumes" Oct 30 15:11:34.157836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:34.157812 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" path="/var/lib/kubelet/pods/52292a7b-4a45-4110-af04-a35cbb09893f/volumes" Oct 30 15:11:34.158390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:34.158364 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" path="/var/lib/kubelet/pods/9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6/volumes" Oct 30 15:11:34.159136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:34.159108 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" path="/var/lib/kubelet/pods/9e5a0723-eb84-458e-a15e-180c90d45e37/volumes" Oct 30 15:11:35.250975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.250933 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9"] Oct 30 15:11:35.252549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252520 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" Oct 30 15:11:35.252549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252548 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252592 6554 cpu_manager.go:414] "containerMap: removing container" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252600 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252643 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252651 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252667 6554 cpu_manager.go:414] "containerMap: removing container" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252674 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252729 6554 cpu_manager.go:414] "containerMap: removing container" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" Oct 30 15:11:35.252735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252736 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" Oct 30 15:11:35.253082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252766 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" Oct 30 15:11:35.253082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252773 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" Oct 30 15:11:35.253082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252795 6554 cpu_manager.go:414] "containerMap: removing container" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" Oct 30 15:11:35.253082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252801 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" Oct 30 15:11:35.253082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252832 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" Oct 30 15:11:35.253082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.252839 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253154 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9143c9e0-e90a-4d0f-8874-2b9a3df5f0f6" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253168 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="4767fe86-4975-44bb-a86f-bafa6b10f05a" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253190 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="154c6b17-aa49-46c2-9cda-7d795d69492f" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253209 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5a0723-eb84-458e-a15e-180c90d45e37" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253226 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="21fe275d-7a84-478c-863d-2db9076587f4" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253236 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc78931-2fc6-42b5-94a1-8a2d98513cf6" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253254 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7a2cff1-a2e4-4c9d-a474-279529d4a336" containerName="istio-proxy" Oct 30 15:11:35.253315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.253266 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="52292a7b-4a45-4110-af04-a35cbb09893f" containerName="istio-proxy" Oct 30 15:11:35.258066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.258037 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.273382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.272332 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9"] Oct 30 15:11:35.281375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.280598 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-216204-serving-cert\"" Oct 30 15:11:35.430191 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.430152 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.430328 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.430270 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.430536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.430516 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.430573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.430549 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxzb6\" (UniqueName: \"kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.531735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.531644 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.531735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.531685 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxzb6\" (UniqueName: \"kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.531991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.531775 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.531991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.531829 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.531991 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:35.531927 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-216204-serving-cert: secret "maas-api-e2e-api-b-216204-serving-cert" not found Oct 30 15:11:35.532167 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:35.531997 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls podName:99094cec-d5d2-4e24-a57c-368572c13477 nodeName:}" failed. No retries permitted until 2026-10-30 15:11:36.031977037 +0000 UTC m=+1399.991018433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls") pod "maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" (UID: "99094cec-d5d2-4e24-a57c-368572c13477") : secret "maas-api-e2e-api-b-216204-serving-cert" not found Oct 30 15:11:35.532622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.532600 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.534320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.534290 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:35.558024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:35.557995 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxzb6\" (UniqueName: \"kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:36.039398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:36.039332 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:36.073369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:36.070544 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\") pod \"maas-api-e2e-api-b-216204-54f54c64c8-qcpr9\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:36.179810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:36.179777 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:36.519740 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:36.519707 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9"] Oct 30 15:11:36.520790 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:11:36.520766 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99094cec_d5d2_4e24_a57c_368572c13477.slice/crio-3cc3351501afa9338b4d3a125adbcdd2f933d14254b0a9da78ba1ed72b33cbb1 WatchSource:0}: Error finding container 3cc3351501afa9338b4d3a125adbcdd2f933d14254b0a9da78ba1ed72b33cbb1: Status 404 returned error can't find the container with id 3cc3351501afa9338b4d3a125adbcdd2f933d14254b0a9da78ba1ed72b33cbb1 Oct 30 15:11:36.642102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:36.642068 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" event={"ID":"99094cec-d5d2-4e24-a57c-368572c13477","Type":"ContainerStarted","Data":"3cc3351501afa9338b4d3a125adbcdd2f933d14254b0a9da78ba1ed72b33cbb1"} Oct 30 15:11:37.659184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:37.659144 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" event={"ID":"99094cec-d5d2-4e24-a57c-368572c13477","Type":"ContainerStarted","Data":"b61999cd6454ec6fe8a8cc4ad74e653f29b10889c3280a89c701d1f4c8142da8"} Oct 30 15:11:37.659716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:37.659278 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:37.707136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:37.707073 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" podStartSLOduration=2.246633533 podStartE2EDuration="2.707054562s" totalImagesPullingTime="460.421029ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:37.705849134 +0000 UTC m=+1401.664890542" watchObservedRunningTime="2026-10-30 15:11:37.707054562 +0000 UTC m=+1401.666095968" Oct 30 15:11:38.542247 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:38.542217 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:11:39.201794 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:39.201759 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:11:39.303099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.303068 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4"] Oct 30 15:11:39.307592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.307569 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.328882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.328860 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4"] Oct 30 15:11:39.336851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.336832 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-24fff1\"" Oct 30 15:11:39.480977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.434708 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-24fff1-dockercfg-zrfg5\"" Oct 30 15:11:39.509532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.484792 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7kch\" (UniqueName: \"kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch\") pod \"payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.519222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.519192 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.524535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.524478 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h"] Oct 30 15:11:39.532493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.532454 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.537423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.537399 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h"] Oct 30 15:11:39.622665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.622636 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7kch\" (UniqueName: \"kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch\") pod \"payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.622934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.622690 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkccl\" (UniqueName: \"kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl\") pod \"payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.622934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.622869 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.623025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.622989 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.623610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.623589 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.657758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.657718 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7kch\" (UniqueName: \"kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch\") pod \"payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.724525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.724477 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkccl\" (UniqueName: \"kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl\") pod \"payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.725028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.724788 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.725652 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.725624 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.751136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.751073 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkccl\" (UniqueName: \"kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl\") pod \"payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.817626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.817595 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b"] Oct 30 15:11:39.823140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.823125 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.839597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.839577 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1b7814-openshift-default-dockercfg-5nltl\"" Oct 30 15:11:39.857244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.857220 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b"] Oct 30 15:11:39.859828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.859805 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:39.923378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.923087 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.927926 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.927988 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928037 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928074 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvpv\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928129 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928312 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928390 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928449 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:39.934373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:39.928614 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.029859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.029822 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.029987 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030037 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030076 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030118 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030149 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvpv\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030200 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030396 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030434 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.030898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.030869 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.031217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.031107 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.034177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.033775 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.034177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.034019 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.034444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.034209 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.051424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.036298 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.112663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.112587 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.114942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.114913 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.115560 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.115538 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvpv\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv\") pod \"e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.152397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.151320 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:40.203389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.196599 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h"] Oct 30 15:11:40.319018 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:11:40.297226 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de2f3cb_6eb4_4028_ac32_f1be0318f8ea.slice/crio-623de9ac197a55b3c90d0372567e82c1d945f9274e0e894de59c40e112700d93 WatchSource:0}: Error finding container 623de9ac197a55b3c90d0372567e82c1d945f9274e0e894de59c40e112700d93: Status 404 returned error can't find the container with id 623de9ac197a55b3c90d0372567e82c1d945f9274e0e894de59c40e112700d93 Oct 30 15:11:40.359784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.358449 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4"] Oct 30 15:11:40.621858 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:11:40.621830 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb468a794_0400_4e15_947a_4f45c2b7bb8f.slice/crio-4f273c0abf64c4150f4578012e888e2b7dfec23a26ac4aff3f69c98df745d43c WatchSource:0}: Error finding container 4f273c0abf64c4150f4578012e888e2b7dfec23a26ac4aff3f69c98df745d43c: Status 404 returned error can't find the container with id 4f273c0abf64c4150f4578012e888e2b7dfec23a26ac4aff3f69c98df745d43c Oct 30 15:11:40.624319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.624283 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b"] Oct 30 15:11:40.625559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.625504 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:11:40.625664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.625588 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:11:40.625664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.625636 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:11:40.690640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.690613 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" event={"ID":"3053b5a0-4454-426b-b2c3-2dad176dfe79","Type":"ContainerStarted","Data":"1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9"} Oct 30 15:11:40.690765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.690661 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" event={"ID":"3053b5a0-4454-426b-b2c3-2dad176dfe79","Type":"ContainerStarted","Data":"16cf8230c23b39d5b52a2901c2bd908cb80d23024705da696785c9f4108dcfba"} Oct 30 15:11:40.690850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.690834 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:40.692102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.692073 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" event={"ID":"b468a794-0400-4e15-947a-4f45c2b7bb8f","Type":"ContainerStarted","Data":"4f273c0abf64c4150f4578012e888e2b7dfec23a26ac4aff3f69c98df745d43c"} Oct 30 15:11:40.693973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.693949 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" event={"ID":"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea","Type":"ContainerStarted","Data":"838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d"} Oct 30 15:11:40.694083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.693990 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" event={"ID":"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea","Type":"ContainerStarted","Data":"623de9ac197a55b3c90d0372567e82c1d945f9274e0e894de59c40e112700d93"} Oct 30 15:11:40.694083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.694039 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:40.740923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.740873 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" podStartSLOduration=1.740828829 podStartE2EDuration="1.740828829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:40.735619659 +0000 UTC m=+1404.694661064" watchObservedRunningTime="2026-10-30 15:11:40.740828829 +0000 UTC m=+1404.699870236" Oct 30 15:11:40.815443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:40.815336 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" podStartSLOduration=1.815313655 podStartE2EDuration="1.815313655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:40.801533048 +0000 UTC m=+1404.760574458" watchObservedRunningTime="2026-10-30 15:11:40.815313655 +0000 UTC m=+1404.774355065" Oct 30 15:11:41.702965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:41.702932 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" event={"ID":"b468a794-0400-4e15-947a-4f45c2b7bb8f","Type":"ContainerStarted","Data":"735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9"} Oct 30 15:11:41.766173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:41.766126 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podStartSLOduration=2.766112969 podStartE2EDuration="2.766112969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:41.764425435 +0000 UTC m=+1405.723466837" watchObservedRunningTime="2026-10-30 15:11:41.766112969 +0000 UTC m=+1405.725154375" Oct 30 15:11:42.156286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:42.156257 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:42.159410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:42.159386 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:42.710254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:42.710217 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:42.711120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:42.711101 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:11:43.673779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:43.673748 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:11:46.175742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.175704 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7"] Oct 30 15:11:46.180269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.180250 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.196891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.196866 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1b7814\"" Oct 30 15:11:46.197049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.196867 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1b7814-dockercfg-wfhc2\"" Oct 30 15:11:46.201219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.201197 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7"] Oct 30 15:11:46.283567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.283506 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt"] Oct 30 15:11:46.334372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.324564 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjnd9\" (UniqueName: \"kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9\") pod \"payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.335774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.335736 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.422584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.422551 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt"] Oct 30 15:11:46.422894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.422873 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.442117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.442081 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjnd9\" (UniqueName: \"kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9\") pod \"payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.442288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.442266 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlgd6\" (UniqueName: \"kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6\") pod \"payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.442393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.442306 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.442393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.442337 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.443814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.443785 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.474457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.474423 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjnd9\" (UniqueName: \"kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9\") pod \"payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.494330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.494297 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:46.544451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.544418 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlgd6\" (UniqueName: \"kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6\") pod \"payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.544649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.544456 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.545812 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.545779 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.578915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.576191 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlgd6\" (UniqueName: \"kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6\") pod \"payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.742153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.742072 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:46.821296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:46.820868 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7"] Oct 30 15:11:47.064737 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:11:47.064712 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467dcb64_f081_4c2e_8cf8_ac6a2dfdbe9b.slice/crio-9c6895498bb0b9f411cc2c2d7a557d94ca4e3023b99d4406582a69081235b1c1 WatchSource:0}: Error finding container 9c6895498bb0b9f411cc2c2d7a557d94ca4e3023b99d4406582a69081235b1c1: Status 404 returned error can't find the container with id 9c6895498bb0b9f411cc2c2d7a557d94ca4e3023b99d4406582a69081235b1c1 Oct 30 15:11:47.065928 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.065902 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt"] Oct 30 15:11:47.777845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.777804 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" event={"ID":"e7a1c21a-b72f-43c9-8142-142465fe33cc","Type":"ContainerStarted","Data":"eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77"} Oct 30 15:11:47.778316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.777867 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" event={"ID":"e7a1c21a-b72f-43c9-8142-142465fe33cc","Type":"ContainerStarted","Data":"285dfdd19bf0aacd89f61c08ec5287fc286e929a9385c62d6cbe5bd927a23c3a"} Oct 30 15:11:47.778316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.777898 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:11:47.779918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.779888 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" event={"ID":"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b","Type":"ContainerStarted","Data":"134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3"} Oct 30 15:11:47.780048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.779929 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" event={"ID":"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b","Type":"ContainerStarted","Data":"9c6895498bb0b9f411cc2c2d7a557d94ca4e3023b99d4406582a69081235b1c1"} Oct 30 15:11:47.780048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.779969 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:47.823663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.823617 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" podStartSLOduration=1.8236046080000001 podStartE2EDuration="1.823604608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:47.82154069 +0000 UTC m=+1411.780582101" watchObservedRunningTime="2026-10-30 15:11:47.823604608 +0000 UTC m=+1411.782646012" Oct 30 15:11:47.868288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:47.868242 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" podStartSLOduration=1.868229451 podStartE2EDuration="1.868229451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:11:47.865750669 +0000 UTC m=+1411.824792076" watchObservedRunningTime="2026-10-30 15:11:47.868229451 +0000 UTC m=+1411.827270855" Oct 30 15:11:51.705757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:51.705723 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:11:51.706201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:51.705800 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:11:54.033954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.033914 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh"] Oct 30 15:11:54.034409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.034180 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" podUID="4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" containerName="maas-api" containerID="cri-o://bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8" gracePeriod=30 Oct 30 15:11:54.411433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.411407 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:11:54.567701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.567666 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj\" (UniqueName: \"kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj\") pod \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " Oct 30 15:11:54.568250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.568215 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") pod \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " Oct 30 15:11:54.568397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.568340 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs\") pod \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " Oct 30 15:11:54.568516 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.568498 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca\") pod \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\" (UID: \"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa\") " Oct 30 15:11:54.568978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.568946 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca" pod "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" (UID: "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:11:54.569385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.569345 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:54.571539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.571506 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls" pod "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" (UID: "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:11:54.572627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.572588 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs" pod "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" (UID: "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:11:54.583603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.583557 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj" pod "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" (UID: "4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa"). InnerVolumeSpecName "kube-api-access-sfdlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:11:54.671432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.671403 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfdlj\" (UniqueName: \"kubernetes.io/projected/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-kube-api-access-sfdlj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:54.671432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.671429 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:54.671607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.671440 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:11:54.855158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.855125 6554 generic.go:356] "Generic (PLEG): container finished" podID="4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" containerID="bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8" exitCode=0 Oct 30 15:11:54.855388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.855178 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" Oct 30 15:11:54.855388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.855209 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" event={"ID":"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa","Type":"ContainerDied","Data":"bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8"} Oct 30 15:11:54.855388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.855265 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh" event={"ID":"4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa","Type":"ContainerDied","Data":"05f060191986d71370aee77fa7748cdc1f108eba88f4cf15edbbfcaa9603ab6b"} Oct 30 15:11:54.855388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.855296 6554 scope.go:122] "RemoveContainer" containerID="bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8" Oct 30 15:11:54.866744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.866720 6554 scope.go:122] "RemoveContainer" containerID="bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8" Oct 30 15:11:54.867015 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:11:54.866994 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8\": container with ID starting with bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8 not found: ID does not exist" containerID="bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8" Oct 30 15:11:54.867062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.867024 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8"} err="failed to get container status \"bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8\": rpc error: code = NotFound desc = could not find container \"bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8\": container with ID starting with bb11307649202ac51a9aa8ce16e33cf121b46e9793b0b55558bff1c09b1bd0c8 not found: ID does not exist" Oct 30 15:11:54.915101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.915076 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh"] Oct 30 15:11:54.928455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:54.928399 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7acff45-68d4447b57-xlsjh"] Oct 30 15:11:56.156712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:56.156680 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" path="/var/lib/kubelet/pods/4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa/volumes" Oct 30 15:11:58.804188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:58.804160 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:11:58.804694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:11:58.804230 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:12:04.028256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.028222 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9"] Oct 30 15:12:04.029160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.029137 6554 cpu_manager.go:414] "containerMap: removing container" podUID="4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" containerName="maas-api" Oct 30 15:12:04.029160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.029158 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" containerName="maas-api" Oct 30 15:12:04.029322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.029312 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bcdae07-f5e1-4de9-81dd-ea3d85dd24aa" containerName="maas-api" Oct 30 15:12:04.034665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.034645 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.051530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.051505 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9"] Oct 30 15:12:04.061156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.061119 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f7acff45\"" Oct 30 15:12:04.061291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.061270 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f7acff45-dockercfg-j5mlb\"" Oct 30 15:12:04.189435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.189331 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb696\" (UniqueName: \"kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.190120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.189627 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.259284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.256381 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2"] Oct 30 15:12:04.268408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.262782 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2"] Oct 30 15:12:04.268408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.262897 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.291238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.290999 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb696\" (UniqueName: \"kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.291238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.291156 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.293229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.292032 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.325209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.325178 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb696\" (UniqueName: \"kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696\") pod \"payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.352482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.352454 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.392637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.392597 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfnj\" (UniqueName: \"kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.392963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.392940 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.495870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.494645 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.495870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.494754 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfnj\" (UniqueName: \"kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.496919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.496859 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.520467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.520439 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfnj\" (UniqueName: \"kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj\") pod \"payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.596625 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.596553 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:04.679345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.679310 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9"] Oct 30 15:12:04.685049 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:12:04.685018 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb644d87e_463d_477e_b3ec_4fb43ef7cea9.slice/crio-79f82ab6410971f67d23882b85979c002464fe644811e80075292dbcc6e7d079 WatchSource:0}: Error finding container 79f82ab6410971f67d23882b85979c002464fe644811e80075292dbcc6e7d079: Status 404 returned error can't find the container with id 79f82ab6410971f67d23882b85979c002464fe644811e80075292dbcc6e7d079 Oct 30 15:12:04.903845 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:12:04.903819 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9278b48d_2b1e_4eb3_8307_56ff68734016.slice/crio-2092e29ab62bb2e033ceb6355ebf2bc1b2f1ad28c8f26be2ffd4ba56c8f60ed1 WatchSource:0}: Error finding container 2092e29ab62bb2e033ceb6355ebf2bc1b2f1ad28c8f26be2ffd4ba56c8f60ed1: Status 404 returned error can't find the container with id 2092e29ab62bb2e033ceb6355ebf2bc1b2f1ad28c8f26be2ffd4ba56c8f60ed1 Oct 30 15:12:04.903947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.903852 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2"] Oct 30 15:12:04.933381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.933343 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" event={"ID":"9278b48d-2b1e-4eb3-8307-56ff68734016","Type":"ContainerStarted","Data":"2092e29ab62bb2e033ceb6355ebf2bc1b2f1ad28c8f26be2ffd4ba56c8f60ed1"} Oct 30 15:12:04.935017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.934992 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" event={"ID":"b644d87e-463d-477e-b3ec-4fb43ef7cea9","Type":"ContainerStarted","Data":"ef1724712191078b81cf9e6927060b9c682b2f52db2a7dadc5040a2bdf260e4f"} Oct 30 15:12:04.935110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.935036 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" event={"ID":"b644d87e-463d-477e-b3ec-4fb43ef7cea9","Type":"ContainerStarted","Data":"79f82ab6410971f67d23882b85979c002464fe644811e80075292dbcc6e7d079"} Oct 30 15:12:04.935246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.935231 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:04.979799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:04.979750 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" podStartSLOduration=0.979734266 podStartE2EDuration="979.734266ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:04.978377321 +0000 UTC m=+1428.937418728" watchObservedRunningTime="2026-10-30 15:12:04.979734266 +0000 UTC m=+1428.938775673" Oct 30 15:12:05.944160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:05.944120 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" event={"ID":"9278b48d-2b1e-4eb3-8307-56ff68734016","Type":"ContainerStarted","Data":"7c995dd6b0c0c77c124b0225279dfeba8ca953fb37780c2d4b9f3f96acedeb8c"} Oct 30 15:12:05.944578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:05.944226 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:05.991079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:05.991032 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" podStartSLOduration=1.991020024 podStartE2EDuration="1.991020024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:05.987609836 +0000 UTC m=+1429.946651241" watchObservedRunningTime="2026-10-30 15:12:05.991020024 +0000 UTC m=+1429.950061554" Oct 30 15:12:09.202441 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:12:09.202406 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:12:15.946889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:15.946858 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:12:16.851198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:16.851164 6554 scope.go:122] "RemoveContainer" containerID="d3ea8a15d1466995e50b5e5c760979502d2a10d7774e6981d7e7110476e7835a" Oct 30 15:12:16.956797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:16.956774 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:12:20.817265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.817235 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp"] Oct 30 15:12:20.824039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.824016 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.842389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.840737 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-216204-openshift-default-dockercfg-ld9nk\"" Oct 30 15:12:20.859117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.859092 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp"] Oct 30 15:12:20.878558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878486 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.878765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878703 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.878843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878788 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.878901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878848 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkf8\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.878901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878891 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.879021 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878926 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.879021 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.878986 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.879137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.879119 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.879194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.879155 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980279 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980315 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980345 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980490 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980529 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980572 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkf8\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980603 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980623 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980678 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.980808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980741 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.981027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.980880 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.981090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.981062 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.981151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.981083 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.981299 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.981276 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.982854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.982834 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:20.982976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:20.982948 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:21.005618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.005583 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:21.006219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.006199 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkf8\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8\") pod \"e2e-api-b-216204-openshift-default-7564567d9c-ng7lp\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:21.140302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.140279 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:21.464425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.464396 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp"] Oct 30 15:12:21.465530 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:12:21.465505 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45ed971_cdbc_4e3f_91ec_6f6f442f0800.slice/crio-0c4958d1bedefbaada542e077abfc288d27c384bc7634ec457c6f25adeb52ce3 WatchSource:0}: Error finding container 0c4958d1bedefbaada542e077abfc288d27c384bc7634ec457c6f25adeb52ce3: Status 404 returned error can't find the container with id 0c4958d1bedefbaada542e077abfc288d27c384bc7634ec457c6f25adeb52ce3 Oct 30 15:12:21.468438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.468401 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:12:21.468540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.468483 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:12:21.468540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:21.468532 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:12:22.074072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:22.074033 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" event={"ID":"e45ed971-cdbc-4e3f-91ec-6f6f442f0800","Type":"ContainerStarted","Data":"2ae82d7cb359284431eec182be8d013da4d9660302fc98e87ef3f2728c6d9cdf"} Oct 30 15:12:22.074447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:22.074131 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" event={"ID":"e45ed971-cdbc-4e3f-91ec-6f6f442f0800","Type":"ContainerStarted","Data":"0c4958d1bedefbaada542e077abfc288d27c384bc7634ec457c6f25adeb52ce3"} Oct 30 15:12:22.141184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:22.141155 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:22.142425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:22.142396 6554 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:12:22.142564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:22.142453 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 15:12:22.153960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:22.153903 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podStartSLOduration=2.1538840280000002 podStartE2EDuration="2.153884028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:22.136521186 +0000 UTC m=+1446.095562593" watchObservedRunningTime="2026-10-30 15:12:22.153884028 +0000 UTC m=+1446.112925436" Oct 30 15:12:23.142031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:23.141987 6554 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:12:23.142598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:23.142066 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 15:12:24.150153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:24.141793 6554 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:12:24.150153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:24.141865 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 15:12:25.140990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:25.140958 6554 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:12:25.141165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:25.141034 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 15:12:26.141325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:26.141290 6554 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:12:26.141779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:26.141363 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 15:12:27.140674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:27.140589 6554 patch_prober.go:29] interesting pod/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 15:12:27.140674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:27.140643 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 15:12:28.155656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:28.155629 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:28.156048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:28.156035 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:28.156774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:28.156752 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:12:39.202879 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:12:39.202842 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:12:47.106378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.106327 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg"] Oct 30 15:12:47.107480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.107447 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" podUID="0316a266-070e-4bdb-875b-86383902d4e5" containerName="maas-api" containerID="cri-o://3203948538b3d026c68b298698eeae17e3fbc336e9843834da197a7e4de03160" gracePeriod=30 Oct 30 15:12:47.310803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.310769 6554 generic.go:356] "Generic (PLEG): container finished" podID="0316a266-070e-4bdb-875b-86383902d4e5" containerID="3203948538b3d026c68b298698eeae17e3fbc336e9843834da197a7e4de03160" exitCode=0 Oct 30 15:12:47.310961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.310845 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" event={"ID":"0316a266-070e-4bdb-875b-86383902d4e5","Type":"ContainerDied","Data":"3203948538b3d026c68b298698eeae17e3fbc336e9843834da197a7e4de03160"} Oct 30 15:12:47.478768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.476751 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.510468 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h\" (UniqueName: \"kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h\") pod \"0316a266-070e-4bdb-875b-86383902d4e5\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.510640 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls\") pod \"0316a266-070e-4bdb-875b-86383902d4e5\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.510870 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca\") pod \"0316a266-070e-4bdb-875b-86383902d4e5\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.510929 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs\") pod \"0316a266-070e-4bdb-875b-86383902d4e5\" (UID: \"0316a266-070e-4bdb-875b-86383902d4e5\") " Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.511944 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca" pod "0316a266-070e-4bdb-875b-86383902d4e5" (UID: "0316a266-070e-4bdb-875b-86383902d4e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.515045 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs" pod "0316a266-070e-4bdb-875b-86383902d4e5" (UID: "0316a266-070e-4bdb-875b-86383902d4e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.515694 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls" pod "0316a266-070e-4bdb-875b-86383902d4e5" (UID: "0316a266-070e-4bdb-875b-86383902d4e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:12:47.522315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.517495 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h" pod "0316a266-070e-4bdb-875b-86383902d4e5" (UID: "0316a266-070e-4bdb-875b-86383902d4e5"). InnerVolumeSpecName "kube-api-access-hf75h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:12:47.617725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.615722 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0316a266-070e-4bdb-875b-86383902d4e5-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:12:47.617725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.615761 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:12:47.617725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.615777 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf75h\" (UniqueName: \"kubernetes.io/projected/0316a266-070e-4bdb-875b-86383902d4e5-kube-api-access-hf75h\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:12:47.617725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:47.615791 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0316a266-070e-4bdb-875b-86383902d4e5-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:12:48.319549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:48.319522 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" Oct 30 15:12:48.320023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:48.319522 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg" event={"ID":"0316a266-070e-4bdb-875b-86383902d4e5","Type":"ContainerDied","Data":"edca2d21ea1c94f5bac81d305cf52b89c19cde33029af3bdf2567dae2c82c235"} Oct 30 15:12:48.320023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:48.319669 6554 scope.go:122] "RemoveContainer" containerID="3203948538b3d026c68b298698eeae17e3fbc336e9843834da197a7e4de03160" Oct 30 15:12:48.380339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:48.380315 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg"] Oct 30 15:12:48.397066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:48.397001 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-7424dbac-697f6c77c4-jlwjg"] Oct 30 15:12:50.158779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:50.158747 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0316a266-070e-4bdb-875b-86383902d4e5" path="/var/lib/kubelet/pods/0316a266-070e-4bdb-875b-86383902d4e5/volumes" Oct 30 15:12:57.233912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.233879 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s"] Oct 30 15:12:57.235426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.235402 6554 cpu_manager.go:414] "containerMap: removing container" podUID="0316a266-070e-4bdb-875b-86383902d4e5" containerName="maas-api" Oct 30 15:12:57.235605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.235582 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0316a266-070e-4bdb-875b-86383902d4e5" containerName="maas-api" Oct 30 15:12:57.235846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.235832 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="0316a266-070e-4bdb-875b-86383902d4e5" containerName="maas-api" Oct 30 15:12:57.245405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.245384 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.247407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.247382 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s"] Oct 30 15:12:57.262099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.262076 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-751ab3-openshift-default-dockercfg-gkstn\"" Oct 30 15:12:57.304223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304193 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304245 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304410 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304554 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304604 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304740 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304724 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304760 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304880 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.304942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.304928 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt4ml\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.406684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.406651 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.406822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.406733 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.406892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.406835 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.406950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.406875 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.406960 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.406991 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407073 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407226 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407285 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt4ml\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407305 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407323 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407421 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407525 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.407566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.407540 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.409314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.409286 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.409829 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.409809 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.438956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.438929 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.439048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.438989 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt4ml\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml\") pod \"e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.572752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.572713 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:57.956205 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:12:57.956167 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87100560_14d1_4906_8d43_90e404d4f607.slice/crio-47c2cce055cd2faee27b59a59420a6c6526a295ef2d28584c099209d6e758534 WatchSource:0}: Error finding container 47c2cce055cd2faee27b59a59420a6c6526a295ef2d28584c099209d6e758534: Status 404 returned error can't find the container with id 47c2cce055cd2faee27b59a59420a6c6526a295ef2d28584c099209d6e758534 Oct 30 15:12:57.956469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.956442 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s"] Oct 30 15:12:57.960230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.960186 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:12:57.960410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.960283 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:12:57.960410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:57.960328 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:12:58.452275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:58.452236 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" event={"ID":"87100560-14d1-4906-8d43-90e404d4f607","Type":"ContainerStarted","Data":"e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2"} Oct 30 15:12:58.452742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:58.452303 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" event={"ID":"87100560-14d1-4906-8d43-90e404d4f607","Type":"ContainerStarted","Data":"47c2cce055cd2faee27b59a59420a6c6526a295ef2d28584c099209d6e758534"} Oct 30 15:12:58.528264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:58.528202 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podStartSLOduration=1.528184899 podStartE2EDuration="1.528184899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:12:58.514155027 +0000 UTC m=+1482.473196434" watchObservedRunningTime="2026-10-30 15:12:58.528184899 +0000 UTC m=+1482.487226306" Oct 30 15:12:58.573794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:58.573767 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:12:58.763575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:58.763488 6554 patch_prober.go:29] interesting pod/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:12:58.763575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:58.763558 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:12:59.580343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:12:59.580314 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:13:00.466804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:00.466775 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:13:00.467814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:00.467797 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:13:01.478747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:01.478668 6554 generic.go:356] "Generic (PLEG): container finished" podID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerID="735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9" exitCode=0 Oct 30 15:13:01.479196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:01.478749 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" event={"ID":"b468a794-0400-4e15-947a-4f45c2b7bb8f","Type":"ContainerDied","Data":"735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9"} Oct 30 15:13:01.479196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:01.479148 6554 scope.go:122] "RemoveContainer" containerID="735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9" Oct 30 15:13:01.480938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:01.480912 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:13:01.481016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:01.480980 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:13:01.481068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:01.481019 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:13:02.495659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.495627 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" event={"ID":"b468a794-0400-4e15-947a-4f45c2b7bb8f","Type":"ContainerStarted","Data":"8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960"} Oct 30 15:13:02.839889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.839810 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k"] Oct 30 15:13:02.844679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.844659 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:02.884709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.884670 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-751ab3\"" Oct 30 15:13:02.885517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.885327 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k"] Oct 30 15:13:02.888853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.888827 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck"] Oct 30 15:13:02.893657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.893628 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:02.916881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.916853 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-751ab3-dockercfg-2g282\"" Oct 30 15:13:02.921704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.921681 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck"] Oct 30 15:13:02.932133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.932105 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-751ab3-c74447484-82n2k\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:02.932249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:02.932232 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhx8j\" (UniqueName: \"kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j\") pod \"payload-processing-e2e-api-a-751ab3-c74447484-82n2k\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:03.039372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.037526 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg9hr\" (UniqueName: \"kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr\") pod \"payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.039372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.037653 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.039372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.037778 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-751ab3-c74447484-82n2k\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:03.039372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.037885 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhx8j\" (UniqueName: \"kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j\") pod \"payload-processing-e2e-api-a-751ab3-c74447484-82n2k\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:03.157372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.156865 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg9hr\" (UniqueName: \"kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr\") pod \"payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.157372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.156968 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.186418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.157973 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.186418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.158057 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:13:03.186418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.158494 6554 patch_prober.go:29] interesting pod/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:13:03.186418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.158536 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 15:13:03.220431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.212325 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-751ab3-c74447484-82n2k\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:03.257370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.255133 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhx8j\" (UniqueName: \"kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j\") pod \"payload-processing-e2e-api-a-751ab3-c74447484-82n2k\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:03.260343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.260312 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg9hr\" (UniqueName: \"kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr\") pod \"payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.486600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.486500 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:03.522342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.522312 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:03.831875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.831846 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k"] Oct 30 15:13:03.832013 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:13:03.831976 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1e1202_396d_46fe_b896_68a4b71d85b9.slice/crio-1d7c20f9da32d4fead5afc7232d2f7a32db393dc261383d14f20de061aa53dc4 WatchSource:0}: Error finding container 1d7c20f9da32d4fead5afc7232d2f7a32db393dc261383d14f20de061aa53dc4: Status 404 returned error can't find the container with id 1d7c20f9da32d4fead5afc7232d2f7a32db393dc261383d14f20de061aa53dc4 Oct 30 15:13:03.884254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:03.884222 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck"] Oct 30 15:13:03.887657 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:13:03.887615 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee72e0a8_acc8_42fd_8351_de68b60cec87.slice/crio-3fe5be37989caed90f58acfb1070407230cd16afaf4f4f5f58663ed13fb00a1b WatchSource:0}: Error finding container 3fe5be37989caed90f58acfb1070407230cd16afaf4f4f5f58663ed13fb00a1b: Status 404 returned error can't find the container with id 3fe5be37989caed90f58acfb1070407230cd16afaf4f4f5f58663ed13fb00a1b Oct 30 15:13:04.176373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.162002 6554 patch_prober.go:29] interesting pod/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:13:04.180966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.180432 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 15:13:04.511735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.511639 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" event={"ID":"9c1e1202-396d-46fe-b896-68a4b71d85b9","Type":"ContainerStarted","Data":"8c385fb76c0e9647fc8c4d73616103eba1f6e957ce74c8110c37a7704ee238b4"} Oct 30 15:13:04.511735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.511705 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" event={"ID":"9c1e1202-396d-46fe-b896-68a4b71d85b9","Type":"ContainerStarted","Data":"1d7c20f9da32d4fead5afc7232d2f7a32db393dc261383d14f20de061aa53dc4"} Oct 30 15:13:04.511954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.511736 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:04.513382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.513324 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" event={"ID":"ee72e0a8-acc8-42fd-8351-de68b60cec87","Type":"ContainerStarted","Data":"4d63448871399cf58fa2240e409fbada04c30dc83eb4e7b980e63cc31fbf60e6"} Oct 30 15:13:04.513492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.513386 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" event={"ID":"ee72e0a8-acc8-42fd-8351-de68b60cec87","Type":"ContainerStarted","Data":"3fe5be37989caed90f58acfb1070407230cd16afaf4f4f5f58663ed13fb00a1b"} Oct 30 15:13:04.513492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.513415 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:04.576372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.573711 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" podStartSLOduration=2.573691974 podStartE2EDuration="2.573691974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:04.572679367 +0000 UTC m=+1488.531720773" watchObservedRunningTime="2026-10-30 15:13:04.573691974 +0000 UTC m=+1488.532733382" Oct 30 15:13:04.619128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:04.619073 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" podStartSLOduration=2.619059281 podStartE2EDuration="2.619059281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:04.617974457 +0000 UTC m=+1488.577015862" watchObservedRunningTime="2026-10-30 15:13:04.619059281 +0000 UTC m=+1488.578100686" Oct 30 15:13:05.179396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:05.174726 6554 patch_prober.go:29] interesting pod/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:13:05.179396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:05.174821 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 15:13:06.178504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:06.178464 6554 patch_prober.go:29] interesting pod/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:13:06.179025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:06.178525 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 15:13:07.158772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:07.158732 6554 patch_prober.go:29] interesting pod/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 15:13:07.159075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:07.159042 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 15:13:08.230907 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:08.230871 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:13:08.231604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:08.231562 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:13:08.232342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:08.232317 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:13:09.204494 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:13:09.204461 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:13:15.523071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:15.523038 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:13:15.523486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:15.523111 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:13:29.901371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.900381 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t"] Oct 30 15:13:29.908977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.908951 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t"] Oct 30 15:13:29.909178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.909102 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:29.941739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.941709 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-47bcfe-dockercfg-hmpd6\"" Oct 30 15:13:29.951793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.951757 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-47bcfe\"" Oct 30 15:13:29.985703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.980463 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6krn6\" (UniqueName: \"kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6\") pod \"payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:29.985703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:29.985123 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.087341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.087311 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6krn6\" (UniqueName: \"kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6\") pod \"payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.087576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.087410 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.088142 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.088119 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.112816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.112768 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6krn6\" (UniqueName: \"kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6\") pod \"payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.228744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.228634 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.606241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.603093 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t"] Oct 30 15:13:30.606241 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:13:30.603806 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae0ba8e_c9d6_4931_813d_05c25f2398b3.slice/crio-31a83ad7bae25072ce822388add507b4a92337a09cc70546bdb70fdade61081b WatchSource:0}: Error finding container 31a83ad7bae25072ce822388add507b4a92337a09cc70546bdb70fdade61081b: Status 404 returned error can't find the container with id 31a83ad7bae25072ce822388add507b4a92337a09cc70546bdb70fdade61081b Oct 30 15:13:30.740445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.740327 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" event={"ID":"bae0ba8e-c9d6-4931-813d-05c25f2398b3","Type":"ContainerStarted","Data":"c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0"} Oct 30 15:13:30.740669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.740645 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:30.740742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:30.740698 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" event={"ID":"bae0ba8e-c9d6-4931-813d-05c25f2398b3","Type":"ContainerStarted","Data":"31a83ad7bae25072ce822388add507b4a92337a09cc70546bdb70fdade61081b"} Oct 30 15:13:39.201517 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:13:39.201484 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:13:41.753603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:41.753560 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:13:41.801387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:41.801313 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" podStartSLOduration=12.801295192 podStartE2EDuration="12.801295192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:13:30.877658337 +0000 UTC m=+1514.836699733" watchObservedRunningTime="2026-10-30 15:13:41.801295192 +0000 UTC m=+1525.760336600" Oct 30 15:13:54.853277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:54.853237 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2"] Oct 30 15:13:54.919203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:54.919160 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9"] Oct 30 15:13:59.853678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:59.853590 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" podUID="9278b48d-2b1e-4eb3-8307-56ff68734016" containerName="payload-processing" containerID="cri-o://7c995dd6b0c0c77c124b0225279dfeba8ca953fb37780c2d4b9f3f96acedeb8c" gracePeriod=25 Oct 30 15:13:59.920307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:13:59.920277 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" podUID="b644d87e-463d-477e-b3ec-4fb43ef7cea9" containerName="payload-pre-processing" containerID="cri-o://ef1724712191078b81cf9e6927060b9c682b2f52db2a7dadc5040a2bdf260e4f" gracePeriod=25 Oct 30 15:14:00.054525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.054489 6554 generic.go:356] "Generic (PLEG): container finished" podID="b644d87e-463d-477e-b3ec-4fb43ef7cea9" containerID="ef1724712191078b81cf9e6927060b9c682b2f52db2a7dadc5040a2bdf260e4f" exitCode=0 Oct 30 15:14:00.054751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.054718 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" event={"ID":"b644d87e-463d-477e-b3ec-4fb43ef7cea9","Type":"ContainerDied","Data":"ef1724712191078b81cf9e6927060b9c682b2f52db2a7dadc5040a2bdf260e4f"} Oct 30 15:14:00.056763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.056735 6554 generic.go:356] "Generic (PLEG): container finished" podID="9278b48d-2b1e-4eb3-8307-56ff68734016" containerID="7c995dd6b0c0c77c124b0225279dfeba8ca953fb37780c2d4b9f3f96acedeb8c" exitCode=0 Oct 30 15:14:00.056884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.056773 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" event={"ID":"9278b48d-2b1e-4eb3-8307-56ff68734016","Type":"ContainerDied","Data":"7c995dd6b0c0c77c124b0225279dfeba8ca953fb37780c2d4b9f3f96acedeb8c"} Oct 30 15:14:00.233464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.233438 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:14:00.257251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.257229 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:14:00.323804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.323771 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume\") pod \"9278b48d-2b1e-4eb3-8307-56ff68734016\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " Oct 30 15:14:00.324011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.323992 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj\" (UniqueName: \"kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj\") pod \"9278b48d-2b1e-4eb3-8307-56ff68734016\" (UID: \"9278b48d-2b1e-4eb3-8307-56ff68734016\") " Oct 30 15:14:00.324118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.324096 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume" pod "9278b48d-2b1e-4eb3-8307-56ff68734016" (UID: "9278b48d-2b1e-4eb3-8307-56ff68734016"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:00.324921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.324897 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9278b48d-2b1e-4eb3-8307-56ff68734016-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:00.326139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.326118 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj" pod "9278b48d-2b1e-4eb3-8307-56ff68734016" (UID: "9278b48d-2b1e-4eb3-8307-56ff68734016"). InnerVolumeSpecName "kube-api-access-xnfnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:00.425403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.425310 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume\") pod \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " Oct 30 15:14:00.425403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.425342 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696\" (UniqueName: \"kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696\") pod \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\" (UID: \"b644d87e-463d-477e-b3ec-4fb43ef7cea9\") " Oct 30 15:14:00.425692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.425668 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume" pod "b644d87e-463d-477e-b3ec-4fb43ef7cea9" (UID: "b644d87e-463d-477e-b3ec-4fb43ef7cea9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:00.426019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.426002 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnfnj\" (UniqueName: \"kubernetes.io/projected/9278b48d-2b1e-4eb3-8307-56ff68734016-kube-api-access-xnfnj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:00.426065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.426022 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b644d87e-463d-477e-b3ec-4fb43ef7cea9-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:00.427484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.427465 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696" pod "b644d87e-463d-477e-b3ec-4fb43ef7cea9" (UID: "b644d87e-463d-477e-b3ec-4fb43ef7cea9"). InnerVolumeSpecName "kube-api-access-jb696". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:00.527585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:00.527561 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb696\" (UniqueName: \"kubernetes.io/projected/b644d87e-463d-477e-b3ec-4fb43ef7cea9-kube-api-access-jb696\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:01.067037 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.066995 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" Oct 30 15:14:01.067832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.067087 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9" event={"ID":"b644d87e-463d-477e-b3ec-4fb43ef7cea9","Type":"ContainerDied","Data":"79f82ab6410971f67d23882b85979c002464fe644811e80075292dbcc6e7d079"} Oct 30 15:14:01.067832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.067153 6554 scope.go:122] "RemoveContainer" containerID="ef1724712191078b81cf9e6927060b9c682b2f52db2a7dadc5040a2bdf260e4f" Oct 30 15:14:01.069710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.069675 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" event={"ID":"9278b48d-2b1e-4eb3-8307-56ff68734016","Type":"ContainerDied","Data":"2092e29ab62bb2e033ceb6355ebf2bc1b2f1ad28c8f26be2ffd4ba56c8f60ed1"} Oct 30 15:14:01.069869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.069766 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2" Oct 30 15:14:01.082848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.082824 6554 scope.go:122] "RemoveContainer" containerID="7c995dd6b0c0c77c124b0225279dfeba8ca953fb37780c2d4b9f3f96acedeb8c" Oct 30 15:14:01.143947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.143916 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9"] Oct 30 15:14:01.158038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.158013 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7acff45-57cc8c848c-7hws9"] Oct 30 15:14:01.188096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.188055 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2"] Oct 30 15:14:01.192875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:01.192851 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7acff45-58f57b54d-hwcd2"] Oct 30 15:14:02.156763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:02.156734 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9278b48d-2b1e-4eb3-8307-56ff68734016" path="/var/lib/kubelet/pods/9278b48d-2b1e-4eb3-8307-56ff68734016/volumes" Oct 30 15:14:02.157175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:02.157161 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b644d87e-463d-477e-b3ec-4fb43ef7cea9" path="/var/lib/kubelet/pods/b644d87e-463d-477e-b3ec-4fb43ef7cea9/volumes" Oct 30 15:14:04.251423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.251388 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f"] Oct 30 15:14:04.252617 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.252588 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b644d87e-463d-477e-b3ec-4fb43ef7cea9" containerName="payload-pre-processing" Oct 30 15:14:04.252617 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.252611 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b644d87e-463d-477e-b3ec-4fb43ef7cea9" containerName="payload-pre-processing" Oct 30 15:14:04.252806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.252623 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9278b48d-2b1e-4eb3-8307-56ff68734016" containerName="payload-processing" Oct 30 15:14:04.252806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.252631 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9278b48d-2b1e-4eb3-8307-56ff68734016" containerName="payload-processing" Oct 30 15:14:04.252911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.252817 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b644d87e-463d-477e-b3ec-4fb43ef7cea9" containerName="payload-pre-processing" Oct 30 15:14:04.252911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.252831 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9278b48d-2b1e-4eb3-8307-56ff68734016" containerName="payload-processing" Oct 30 15:14:04.260206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.260179 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.280924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.278795 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f"] Oct 30 15:14:04.379572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379543 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379576 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379698 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379734 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379752 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379854 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379890 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.379932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.379925 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.380044 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.380025 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzsv\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481162 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481198 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481226 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481269 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzsv\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481413 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481435 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481498 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481536 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481562 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481670 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481795 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.481908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481879 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.482017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.481994 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.482305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.482283 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.483761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.483732 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.483921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.483898 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.508061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.508014 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.508506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.508483 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzsv\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.582936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.582915 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:04.959399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.957685 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f"] Oct 30 15:14:04.963693 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:14:04.962941 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351672e0_ce2f_4ea0_8987_0b2471da2f6b.slice/crio-ef15e8faa472684e616a26c547d1fb9c46873b77f84ecf032c02ae97039778eb WatchSource:0}: Error finding container ef15e8faa472684e616a26c547d1fb9c46873b77f84ecf032c02ae97039778eb: Status 404 returned error can't find the container with id ef15e8faa472684e616a26c547d1fb9c46873b77f84ecf032c02ae97039778eb Oct 30 15:14:04.966550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.966510 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:04.966664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.966613 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:04.966664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:04.966649 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:05.127177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:05.127146 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" event={"ID":"351672e0-ce2f-4ea0-8987-0b2471da2f6b","Type":"ContainerStarted","Data":"f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a"} Oct 30 15:14:05.127378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:05.127204 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" event={"ID":"351672e0-ce2f-4ea0-8987-0b2471da2f6b","Type":"ContainerStarted","Data":"ef15e8faa472684e616a26c547d1fb9c46873b77f84ecf032c02ae97039778eb"} Oct 30 15:14:05.180265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:05.180209 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podStartSLOduration=1.18019553 podStartE2EDuration="1.18019553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:05.178229953 +0000 UTC m=+1549.137271359" watchObservedRunningTime="2026-10-30 15:14:05.18019553 +0000 UTC m=+1549.139236978" Oct 30 15:14:05.583287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:05.583252 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:05.991372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:05.987137 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n"] Oct 30 15:14:06.002419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:05.993454 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018466 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018604 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018692 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018743 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppb2\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018782 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018865 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.018966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.018948 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.019135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.019053 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.019135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.019081 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.026940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.026912 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n"] Oct 30 15:14:06.031333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.031306 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a65941-openshift-default-dockercfg-6hn7v\"" Oct 30 15:14:06.120701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120668 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.120878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120716 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppb2\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.120878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120753 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.120878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120808 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.120878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120864 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120934 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.120954 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121101 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121164 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121404 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121553 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121699 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121713 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.121917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.121837 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.123755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.123726 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.123920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.123900 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.338472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.335013 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.389425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.351248 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppb2\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2\") pod \"e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.404755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.404718 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:06.587407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.585391 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 15:14:06.587407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.585456 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 15:14:06.818662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.818620 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n"] Oct 30 15:14:06.833923 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:14:06.833250 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab402479_669a_4335_8c47_b7bce678d1d8.slice/crio-66a0b6ea8b496d73fcf34a3eba1a682e60528476767dc0f94dce8600c704257e WatchSource:0}: Error finding container 66a0b6ea8b496d73fcf34a3eba1a682e60528476767dc0f94dce8600c704257e: Status 404 returned error can't find the container with id 66a0b6ea8b496d73fcf34a3eba1a682e60528476767dc0f94dce8600c704257e Oct 30 15:14:06.840369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.840207 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:06.840369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.840311 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:06.840576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:06.840410 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:07.148647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.148614 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" event={"ID":"ab402479-669a-4335-8c47-b7bce678d1d8","Type":"ContainerStarted","Data":"d8ab73cdd61ed7866151621d541b96bc3cee4ceec1d201b46b9d4aa1fda759fb"} Oct 30 15:14:07.148855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.148663 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" event={"ID":"ab402479-669a-4335-8c47-b7bce678d1d8","Type":"ContainerStarted","Data":"66a0b6ea8b496d73fcf34a3eba1a682e60528476767dc0f94dce8600c704257e"} Oct 30 15:14:07.209708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.209651 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podStartSLOduration=2.209637052 podStartE2EDuration="2.209637052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:07.204119759 +0000 UTC m=+1551.163161175" watchObservedRunningTime="2026-10-30 15:14:07.209637052 +0000 UTC m=+1551.168678522" Oct 30 15:14:07.408511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.408424 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:07.410047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.410018 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:14:07.410162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.410065 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:14:07.583582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.583553 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 15:14:07.583769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:07.583598 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 15:14:08.413332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:08.413297 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:08.584188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:08.584154 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 15:14:08.584381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:08.584206 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 15:14:09.185598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:09.185559 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:09.186711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:09.186689 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:14:09.207000 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:14:09.206968 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:14:09.584413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:09.584310 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 15:14:09.584413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:09.584392 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 15:14:10.584248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:10.584209 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 15:14:10.584472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:10.584276 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 15:14:11.583654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:11.583622 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 15:14:11.583859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:11.583681 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 15:14:12.588180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:12.588151 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:12.588655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:12.588642 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:12.589450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:12.589432 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:14:16.946644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:16.946610 6554 scope.go:122] "RemoveContainer" containerID="0806c19cfec5b224271a3f0d09be2580aa20a6653439e7661393830adb4f009b" Oct 30 15:14:18.352435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.352398 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk"] Oct 30 15:14:18.401707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.401671 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk"] Oct 30 15:14:18.404607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.404568 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.423970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.423945 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-47bcfe-serving-cert\"" Oct 30 15:14:18.505141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.505085 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.505345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.505281 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.505530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.505510 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.505609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.505579 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmc6b\" (UniqueName: \"kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.608173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.608034 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.608506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.608481 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmc6b\" (UniqueName: \"kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.609002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.608974 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.609240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.609224 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.611010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.610983 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.611142 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:14:18.611091 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-serving-cert: secret "maas-api-e2e-api-b-47bcfe-serving-cert" not found Oct 30 15:14:18.611211 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:14:18.611152 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls podName:b88e57bc-a2e9-494a-a7fd-924b957fef1c nodeName:}" failed. No retries permitted until 2026-10-30 15:14:19.111133034 +0000 UTC m=+1563.070174431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls") pod "maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" (UID: "b88e57bc-a2e9-494a-a7fd-924b957fef1c") : secret "maas-api-e2e-api-b-47bcfe-serving-cert" not found Oct 30 15:14:18.619257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.619235 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:18.639687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:18.639650 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmc6b\" (UniqueName: \"kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:19.120376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.119676 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:19.155374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.146794 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\") pod \"maas-api-e2e-api-b-47bcfe-5448f67744-pwntk\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:19.298489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.298455 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr"] Oct 30 15:14:19.307667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.307601 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.313250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.313221 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr"] Oct 30 15:14:19.331098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.331072 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:19.432334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.432299 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hcx\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.432938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.432912 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.433125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433110 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.433248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433235 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.433416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433401 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.433536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433523 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.433672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433655 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.433794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433779 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.434012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.433997 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542447 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542497 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542564 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542599 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542641 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542674 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542791 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542954 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hcx\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.542985 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.543607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.543463 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.546621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.545901 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.546749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.546627 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.546820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.546781 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.548952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.547502 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.548952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.547974 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.551271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.551236 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.615811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.607028 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hcx\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.627705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.627632 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.635526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.635325 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:19.800016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:19.799987 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk"] Oct 30 15:14:19.807846 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:14:19.807819 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88e57bc_a2e9_494a_a7fd_924b957fef1c.slice/crio-6a5db9a39b2d2d1d781efa004dd21afd4c8503202575d6a7172c47a08fb76ab5 WatchSource:0}: Error finding container 6a5db9a39b2d2d1d781efa004dd21afd4c8503202575d6a7172c47a08fb76ab5: Status 404 returned error can't find the container with id 6a5db9a39b2d2d1d781efa004dd21afd4c8503202575d6a7172c47a08fb76ab5 Oct 30 15:14:20.019981 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.019572 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr"] Oct 30 15:14:20.021180 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:14:20.021156 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b96720_1681_466e_b94d_07bc26a00523.slice/crio-c23651c28de41b67c7e6fd8960d30aa3a34527a974cebba393f41f52da21efe9 WatchSource:0}: Error finding container c23651c28de41b67c7e6fd8960d30aa3a34527a974cebba393f41f52da21efe9: Status 404 returned error can't find the container with id c23651c28de41b67c7e6fd8960d30aa3a34527a974cebba393f41f52da21efe9 Oct 30 15:14:20.024492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.024459 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:20.024583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.024543 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:20.024718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.024699 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:20.484383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.482280 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" event={"ID":"b88e57bc-a2e9-494a-a7fd-924b957fef1c","Type":"ContainerStarted","Data":"a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41"} Oct 30 15:14:20.484383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.482431 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" event={"ID":"b88e57bc-a2e9-494a-a7fd-924b957fef1c","Type":"ContainerStarted","Data":"6a5db9a39b2d2d1d781efa004dd21afd4c8503202575d6a7172c47a08fb76ab5"} Oct 30 15:14:20.484383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.483635 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:20.486168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.486131 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" event={"ID":"f6b96720-1681-466e-b94d-07bc26a00523","Type":"ContainerStarted","Data":"e526ad9992c1e05a1d7fd4f5ef53735f7e0c983885737e00588bc08cd10f531f"} Oct 30 15:14:20.486311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.486182 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" event={"ID":"f6b96720-1681-466e-b94d-07bc26a00523","Type":"ContainerStarted","Data":"c23651c28de41b67c7e6fd8960d30aa3a34527a974cebba393f41f52da21efe9"} Oct 30 15:14:20.532205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.532094 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" podStartSLOduration=2.170630734 podStartE2EDuration="2.532073752s" totalImagesPullingTime="361.443018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:20.530128342 +0000 UTC m=+1564.489169748" watchObservedRunningTime="2026-10-30 15:14:20.532073752 +0000 UTC m=+1564.491115158" Oct 30 15:14:20.589372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.589089 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podStartSLOduration=1.5890721989999999 podStartE2EDuration="1.589072199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:20.587701727 +0000 UTC m=+1564.546743129" watchObservedRunningTime="2026-10-30 15:14:20.589072199 +0000 UTC m=+1564.548113604" Oct 30 15:14:20.636157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:20.636119 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:21.638734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:21.638700 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:21.639127 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:21.638753 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 15:14:22.639193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:22.639135 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:22.639820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:22.639783 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 15:14:23.647872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:23.647835 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:23.648483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:23.648451 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 15:14:24.636782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:24.636738 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:24.637067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:24.637025 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 15:14:25.637853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:25.637417 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:25.637853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:25.637477 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 15:14:26.636999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:26.636969 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 30 15:14:26.637172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:26.637023 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 30 15:14:27.578658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:27.578627 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:14:28.096524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:28.096488 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:28.097152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:28.097123 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:28.099155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:28.099123 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:14:30.467853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.467823 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:30.468296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.467883 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:30.713968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.713925 6554 generic.go:356] "Generic (PLEG): container finished" podID="87100560-14d1-4906-8d43-90e404d4f607" containerID="e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2" exitCode=0 Oct 30 15:14:30.714169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.714026 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" event={"ID":"87100560-14d1-4906-8d43-90e404d4f607","Type":"ContainerDied","Data":"e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2"} Oct 30 15:14:30.714552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.714526 6554 scope.go:122] "RemoveContainer" containerID="e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2" Oct 30 15:14:30.718964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.718900 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:30.719061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.718973 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:30.719061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:30.719007 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:31.723395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:31.723339 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" event={"ID":"87100560-14d1-4906-8d43-90e404d4f607","Type":"ContainerStarted","Data":"e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf"} Oct 30 15:14:32.579156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:32.579069 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:14:32.584883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:32.584481 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:32.584883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:32.584546 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:33.577754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:33.577717 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:33.578390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:33.578332 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:34.284988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.284951 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255"] Oct 30 15:14:34.291829 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.291805 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.300196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.300169 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255"] Oct 30 15:14:34.410059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410028 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtfv\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410079 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410147 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410235 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410358 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410403 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410441 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410583 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.410641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.410631 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.511895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.511865 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.511895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.511899 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.511927 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512035 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512097 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512151 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512245 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512300 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512386 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512448 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtfv\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512563 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512717 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512796 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.512887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.512864 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.514455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.514434 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.514576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.514560 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.537016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.536958 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.537111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.537068 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtfv\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv\") pod \"e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.578371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.574852 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:34.578371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.574932 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:34.617386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.617363 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:34.983004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.982980 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255"] Oct 30 15:14:34.983988 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:14:34.983961 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6d4be6_01e2_4bcb_ad1e_8c5fba1f6f6c.slice/crio-a5113a95d9e4414756d81ee98e4aabda4c948a161f1693a402e313836b786deb WatchSource:0}: Error finding container a5113a95d9e4414756d81ee98e4aabda4c948a161f1693a402e313836b786deb: Status 404 returned error can't find the container with id a5113a95d9e4414756d81ee98e4aabda4c948a161f1693a402e313836b786deb Oct 30 15:14:34.987167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.987125 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:34.987288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.987203 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:34.987288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:34.987234 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:14:35.576371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:35.575704 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:35.576371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:35.575762 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:35.793333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:35.793294 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" event={"ID":"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c","Type":"ContainerStarted","Data":"8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506"} Oct 30 15:14:35.793845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:35.793375 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" event={"ID":"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c","Type":"ContainerStarted","Data":"a5113a95d9e4414756d81ee98e4aabda4c948a161f1693a402e313836b786deb"} Oct 30 15:14:35.894409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:35.894359 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podStartSLOduration=1.8943336899999998 podStartE2EDuration="1.89433369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:35.876734732 +0000 UTC m=+1579.835776149" watchObservedRunningTime="2026-10-30 15:14:35.89433369 +0000 UTC m=+1579.853375095" Oct 30 15:14:36.577371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:36.574111 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:36.577371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:36.574174 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:36.617743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:36.617710 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:36.619570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:36.619531 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 15:14:36.619697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:36.619587 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 15:14:37.575377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:37.574023 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 15:14:37.575377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:37.574081 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 15:14:37.618334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:37.618307 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 15:14:37.618490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:37.618383 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 15:14:38.584070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:38.584037 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:14:38.584595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:38.584454 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:14:38.585144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:38.585122 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:14:38.618317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:38.618289 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 15:14:38.618434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:38.618333 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 15:14:39.206497 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:14:39.206458 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:14:39.618687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:39.618609 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 15:14:39.618687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:39.618660 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 15:14:40.618247 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:40.618217 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 15:14:40.618486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:40.618272 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 15:14:41.289409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:41.284504 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms"] Oct 30 15:14:41.289409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:41.284836 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" containerID="cri-o://b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d" gracePeriod=30 Oct 30 15:14:41.618717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:41.618631 6554 patch_prober.go:29] interesting pod/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body= Oct 30 15:14:41.618717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:41.618685 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" Oct 30 15:14:42.623083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:42.623054 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:42.623548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:42.623460 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:42.624057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:42.624038 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:14:46.664136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.664102 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:14:46.775404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.775360 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.775595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.775452 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.775672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.775601 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.775809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.775779 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.775888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.775817 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.775948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.775907 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.776203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.776027 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.776203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.776064 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.776203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.776118 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy\") pod \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\" (UID: \"9c397e63-4528-4cbd-8ad0-30b693ca98f8\") " Oct 30 15:14:46.776538 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.776494 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.777002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.776977 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.777232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.777210 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.777534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.777511 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.777898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.777876 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:14:46.778128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.778109 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.778288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.778262 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:46.778884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.778858 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:14:46.779114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.779094 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:14:46.780658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.780629 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm" pod "9c397e63-4528-4cbd-8ad0-30b693ca98f8" (UID: "9c397e63-4528-4cbd-8ad0-30b693ca98f8"). InnerVolumeSpecName "kube-api-access-pltrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878847 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878874 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878883 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878892 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pltrm\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-kube-api-access-pltrm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878902 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878910 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.878915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878918 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c397e63-4528-4cbd-8ad0-30b693ca98f8-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.879234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.878926 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c397e63-4528-4cbd-8ad0-30b693ca98f8-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:14:46.993979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.993948 6554 generic.go:356] "Generic (PLEG): container finished" podID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerID="b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d" exitCode=0 Oct 30 15:14:46.994176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.994006 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" Oct 30 15:14:46.994176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.994033 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" event={"ID":"9c397e63-4528-4cbd-8ad0-30b693ca98f8","Type":"ContainerDied","Data":"b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d"} Oct 30 15:14:46.994176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.994086 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms" event={"ID":"9c397e63-4528-4cbd-8ad0-30b693ca98f8","Type":"ContainerDied","Data":"560df032776b2325dbd1577266233a309bf8009c07f2a2ed2884b0b8b212dd35"} Oct 30 15:14:46.994176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:46.994109 6554 scope.go:122] "RemoveContainer" containerID="b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d" Oct 30 15:14:47.006366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:47.006329 6554 scope.go:122] "RemoveContainer" containerID="b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d" Oct 30 15:14:47.006638 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:14:47.006619 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d\": container with ID starting with b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d not found: ID does not exist" containerID="b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d" Oct 30 15:14:47.006693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:47.006646 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d"} err="failed to get container status \"b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d\": rpc error: code = NotFound desc = could not find container \"b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d\": container with ID starting with b5b3978b4ea09f561f630ac50c31e2c6c24661063317a0aaabb9a64922a3ee2d not found: ID does not exist" Oct 30 15:14:47.055125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:47.055095 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms"] Oct 30 15:14:47.060659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:47.060638 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-rjzms"] Oct 30 15:14:48.200547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:48.199469 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" path="/var/lib/kubelet/pods/9c397e63-4528-4cbd-8ad0-30b693ca98f8/volumes" Oct 30 15:14:51.231207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.231164 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn"] Oct 30 15:14:51.232233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.232209 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" Oct 30 15:14:51.232233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.232233 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" Oct 30 15:14:51.232483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.232467 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c397e63-4528-4cbd-8ad0-30b693ca98f8" containerName="istio-proxy" Oct 30 15:14:51.237742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.237720 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.257922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.257899 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn"] Oct 30 15:14:51.266916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.266891 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-496acb-dockercfg-n2jkt\"" Oct 30 15:14:51.266916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.266900 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-496acb\"" Oct 30 15:14:51.267250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.267222 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkc6\" (UniqueName: \"kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6\") pod \"payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.267501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.267472 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.375410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.371667 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkc6\" (UniqueName: \"kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6\") pod \"payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.375410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.371872 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.375410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.372913 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.489550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.475791 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkc6\" (UniqueName: \"kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6\") pod \"payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.589238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.572244 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:51.908521 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:14:51.908492 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88fd47f8_b0f8_4485_bb2d_d10db66652a7.slice/crio-8465cc31df79fa5a968f967b2751f99690e8ebeffd5fbc2380607d019f6b05be WatchSource:0}: Error finding container 8465cc31df79fa5a968f967b2751f99690e8ebeffd5fbc2380607d019f6b05be: Status 404 returned error can't find the container with id 8465cc31df79fa5a968f967b2751f99690e8ebeffd5fbc2380607d019f6b05be Oct 30 15:14:51.908765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:51.908741 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn"] Oct 30 15:14:52.063193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:52.063163 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" event={"ID":"88fd47f8-b0f8-4485-bb2d-d10db66652a7","Type":"ContainerStarted","Data":"22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d"} Oct 30 15:14:52.063316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:52.063220 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" event={"ID":"88fd47f8-b0f8-4485-bb2d-d10db66652a7","Type":"ContainerStarted","Data":"8465cc31df79fa5a968f967b2751f99690e8ebeffd5fbc2380607d019f6b05be"} Oct 30 15:14:52.063316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:52.063256 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:14:52.111763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:52.111719 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" podStartSLOduration=1.111706288 podStartE2EDuration="1.111706288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:14:52.111476913 +0000 UTC m=+1596.070518318" watchObservedRunningTime="2026-10-30 15:14:52.111706288 +0000 UTC m=+1596.070747692" Oct 30 15:14:57.447937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:14:57.447905 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl"] Oct 30 15:15:00.208869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.208833 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj"] Oct 30 15:15:00.214501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.214476 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c"] Oct 30 15:15:00.214639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.214611 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.222757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.222733 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.236825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.236802 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss"] Oct 30 15:15:00.248391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.248338 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj"] Oct 30 15:15:00.248529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.248442 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.251408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.251384 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c"] Oct 30 15:15:00.256802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.256405 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp"] Oct 30 15:15:00.264837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.264813 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.267695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.267674 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss"] Oct 30 15:15:00.273088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.273063 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22"] Oct 30 15:15:00.285077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.285056 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:15:00.297055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.291922 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw"] Oct 30 15:15:00.297055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.292581 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.323879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.315863 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp"] Oct 30 15:15:00.323879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.316026 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt"] Oct 30 15:15:00.323879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.315957 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.325886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.325862 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.326071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.325778 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22"] Oct 30 15:15:00.326179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.326108 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b"] Oct 30 15:15:00.341862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.341839 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql"] Oct 30 15:15:00.352013 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.351954 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.354090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354061 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.354207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354126 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmjq\" (UniqueName: \"kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq\") pod \"maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.354207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354172 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgm5\" (UniqueName: \"kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.354330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354210 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj6td\" (UniqueName: \"kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td\") pod \"maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.354330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354256 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhsjq\" (UniqueName: \"kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.354469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354335 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.354529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354454 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.354588 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354556 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbkch\" (UniqueName: \"kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.354646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354599 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.354646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.354637 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.368541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.368519 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt"] Oct 30 15:15:00.368631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.368563 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw"] Oct 30 15:15:00.368631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.368592 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b"] Oct 30 15:15:00.368631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.368619 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql"] Oct 30 15:15:00.368744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.368637 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.456027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456002 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbkch\" (UniqueName: \"kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.456154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456042 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.456154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456071 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.456154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456107 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9vgr\" (UniqueName: \"kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr\") pod \"maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.456154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456126 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbvk\" (UniqueName: \"kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.456314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456161 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.456404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456378 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.456544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456523 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.456583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456560 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tw55\" (UniqueName: \"kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55\") pod \"maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.456713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456696 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.456772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456756 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmjq\" (UniqueName: \"kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq\") pod \"maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.456810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456769 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.456810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456801 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.456887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456811 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgm5\" (UniqueName: \"kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.456887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456862 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj6td\" (UniqueName: \"kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td\") pod \"maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.456971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456917 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhsjq\" (UniqueName: \"kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.457014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.456966 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.457050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.457040 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.457121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.457101 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzj5p\" (UniqueName: \"kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.457221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.457201 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.457430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.457409 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.457653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.457633 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.457884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.457864 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.481767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.481711 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbkch\" (UniqueName: \"kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch\") pod \"maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.481978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.481953 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmjq\" (UniqueName: \"kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq\") pod \"maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.482264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.482243 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgm5\" (UniqueName: \"kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.482842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.482823 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhsjq\" (UniqueName: \"kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.483158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.483141 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj6td\" (UniqueName: \"kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td\") pod \"maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.555193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.555171 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:00.558583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.558562 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.558683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.558594 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tw55\" (UniqueName: \"kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55\") pod \"maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.558739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.558690 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.558800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.558749 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzj5p\" (UniqueName: \"kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.558902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.558883 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9vgr\" (UniqueName: \"kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr\") pod \"maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.558996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.558929 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbvk\" (UniqueName: \"kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.559112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.559089 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.559187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.559143 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.560537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.559843 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.560537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.559993 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.560537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.560026 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.560537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.560317 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.572880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.572859 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:00.584014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.583976 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tw55\" (UniqueName: \"kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55\") pod \"maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.584696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.584651 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9vgr\" (UniqueName: \"kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr\") pod \"maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.584862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.584815 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzj5p\" (UniqueName: \"kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.585394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.585342 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbvk\" (UniqueName: \"kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.592896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.592873 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:00.623802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.621060 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:00.633532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.632819 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:00.673188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.673157 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:00.673763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.673742 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:00.688629 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.688604 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:00.695496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.695286 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:00.906402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:00.904820 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj"] Oct 30 15:15:00.916083 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:00.916035 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7edcdb2_552f_4a1c_b36f_dfee06683616.slice/crio-72612c99e8d6901fb47b63fa15dd136021326aaf3da601991ffb4aa8683d32ef WatchSource:0}: Error finding container 72612c99e8d6901fb47b63fa15dd136021326aaf3da601991ffb4aa8683d32ef: Status 404 returned error can't find the container with id 72612c99e8d6901fb47b63fa15dd136021326aaf3da601991ffb4aa8683d32ef Oct 30 15:15:01.116478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.116447 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c"] Oct 30 15:15:01.184139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.184078 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss"] Oct 30 15:15:01.202287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.202244 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerStarted","Data":"72612c99e8d6901fb47b63fa15dd136021326aaf3da601991ffb4aa8683d32ef"} Oct 30 15:15:01.246268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.246239 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp"] Oct 30 15:15:01.252265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.252235 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22"] Oct 30 15:15:01.252946 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:01.252923 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568 WatchSource:0}: Error finding container 10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568: Status 404 returned error can't find the container with id 10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568 Oct 30 15:15:01.255617 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:01.255581 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640e515c_16b2_49d4_beea_43ee099f7f09.slice/crio-d4be26719086f206a4052cf5e8953e48fb1fee38ee06470828d603d6c489b6f3 WatchSource:0}: Error finding container d4be26719086f206a4052cf5e8953e48fb1fee38ee06470828d603d6c489b6f3: Status 404 returned error can't find the container with id d4be26719086f206a4052cf5e8953e48fb1fee38ee06470828d603d6c489b6f3 Oct 30 15:15:01.325448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.325414 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt"] Oct 30 15:15:01.325843 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:01.325808 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e987c8_a676_4474_94ad_89b06c778051.slice/crio-c64da047df038dfb3f2eaeb1067c27cf9087e3b7a246d7f920f0b2e3f4461130 WatchSource:0}: Error finding container c64da047df038dfb3f2eaeb1067c27cf9087e3b7a246d7f920f0b2e3f4461130: Status 404 returned error can't find the container with id c64da047df038dfb3f2eaeb1067c27cf9087e3b7a246d7f920f0b2e3f4461130 Oct 30 15:15:01.327053 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:01.327031 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c26b7b_085b_424e_abe3_5111cb128b1a.slice/crio-30758e6d1e51b8f168e59c838abc1e6ab3fc5c04fe8644114e7b01851723c47c WatchSource:0}: Error finding container 30758e6d1e51b8f168e59c838abc1e6ab3fc5c04fe8644114e7b01851723c47c: Status 404 returned error can't find the container with id 30758e6d1e51b8f168e59c838abc1e6ab3fc5c04fe8644114e7b01851723c47c Oct 30 15:15:01.330153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.330130 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw"] Oct 30 15:15:01.340703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.340680 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b"] Oct 30 15:15:01.344778 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:01.344753 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048b6a4e_fbee_489e_a415_698df68d4bb5.slice/crio-066c145b1567e7ded34bffb66dd62ee039cbed3301aeb1d90f8f36c5f078dd0d WatchSource:0}: Error finding container 066c145b1567e7ded34bffb66dd62ee039cbed3301aeb1d90f8f36c5f078dd0d: Status 404 returned error can't find the container with id 066c145b1567e7ded34bffb66dd62ee039cbed3301aeb1d90f8f36c5f078dd0d Oct 30 15:15:01.393368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:01.393328 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql"] Oct 30 15:15:01.563708 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:01.563650 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice/crio-1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af WatchSource:0}: Error finding container 1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af: Status 404 returned error can't find the container with id 1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af Oct 30 15:15:02.213030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.212995 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerStarted","Data":"46a017dbed3a705ad0380b20072f0d0187a32fed5deb9b39e84bd47709394f07"} Oct 30 15:15:02.213188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.213046 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerStarted","Data":"c64da047df038dfb3f2eaeb1067c27cf9087e3b7a246d7f920f0b2e3f4461130"} Oct 30 15:15:02.214342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.214318 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" event={"ID":"048b6a4e-fbee-489e-a415-698df68d4bb5","Type":"ContainerStarted","Data":"066c145b1567e7ded34bffb66dd62ee039cbed3301aeb1d90f8f36c5f078dd0d"} Oct 30 15:15:02.216036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.216012 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerStarted","Data":"4e73f42d072a86676fffdc31cf241bb2966de9884321a4c19bba9d916d49731a"} Oct 30 15:15:02.217842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.217816 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerStarted","Data":"88e333f798570920db69d1932a8809348763f9af155b2394d6d2f991ca04cd2a"} Oct 30 15:15:02.217951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.217856 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerStarted","Data":"eb2e320b32b122a26af89e477d31162630f86f4f017e57af2b3b4335c6490bef"} Oct 30 15:15:02.219654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.219624 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerStarted","Data":"006d25cf782add05135911df608585d1440099907c2fc65d60f97a26ee6009ce"} Oct 30 15:15:02.219749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.219673 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerStarted","Data":"10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568"} Oct 30 15:15:02.221268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.221241 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" event={"ID":"b8c26b7b-085b-424e-abe3-5111cb128b1a","Type":"ContainerStarted","Data":"a00be06f7b3e9b34b8e59ffd884f82d106ab139a3a48a27a9897e457217c43ef"} Oct 30 15:15:02.221387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.221290 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" event={"ID":"b8c26b7b-085b-424e-abe3-5111cb128b1a","Type":"ContainerStarted","Data":"30758e6d1e51b8f168e59c838abc1e6ab3fc5c04fe8644114e7b01851723c47c"} Oct 30 15:15:02.222642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.222620 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerStarted","Data":"1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af"} Oct 30 15:15:02.224315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.224286 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerStarted","Data":"bf784696ace29def89d15c1eab0884ecde007a7646c4de20456c0217fc707290"} Oct 30 15:15:02.224453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.224328 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerStarted","Data":"d4be26719086f206a4052cf5e8953e48fb1fee38ee06470828d603d6c489b6f3"} Oct 30 15:15:02.225855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.225834 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerStarted","Data":"2c4e92b9152903f47a8ba7be54601eeb9cbf2d742f4faf0e363ec2e90a5f0e72"} Oct 30 15:15:02.225935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.225875 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerStarted","Data":"f8e11c9d58e9eb861adecc6376dde46c00572df712fdaa80758550001c02d798"} Oct 30 15:15:02.258875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.258825 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" podStartSLOduration=2.258809173 podStartE2EDuration="2.258809173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.258402629 +0000 UTC m=+1606.217444034" watchObservedRunningTime="2026-10-30 15:15:02.258809173 +0000 UTC m=+1606.217850580" Oct 30 15:15:02.304455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.304416 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" podStartSLOduration=2.304403136 podStartE2EDuration="2.304403136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.301873091 +0000 UTC m=+1606.260914496" watchObservedRunningTime="2026-10-30 15:15:02.304403136 +0000 UTC m=+1606.263444540" Oct 30 15:15:02.347392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.347291 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" podStartSLOduration=2.347274677 podStartE2EDuration="2.347274677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.345649623 +0000 UTC m=+1606.304691026" watchObservedRunningTime="2026-10-30 15:15:02.347274677 +0000 UTC m=+1606.306316084" Oct 30 15:15:02.392561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.392519 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" podStartSLOduration=2.392505624 podStartE2EDuration="2.392505624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.388732302 +0000 UTC m=+1606.347773711" watchObservedRunningTime="2026-10-30 15:15:02.392505624 +0000 UTC m=+1606.351547029" Oct 30 15:15:02.432000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.431962 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" podStartSLOduration=2.431948823 podStartE2EDuration="2.431948823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.430910044 +0000 UTC m=+1606.389951450" watchObservedRunningTime="2026-10-30 15:15:02.431948823 +0000 UTC m=+1606.390990224" Oct 30 15:15:02.448808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.448779 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" podUID="6c1aa13d-72fd-4e10-9733-4773fc7c69b5" containerName="payload-pre-processing" containerID="cri-o://be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f" gracePeriod=25 Oct 30 15:15:02.476937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.476890 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" podStartSLOduration=2.476875143 podStartE2EDuration="2.476875143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.472922257 +0000 UTC m=+1606.431963662" watchObservedRunningTime="2026-10-30 15:15:02.476875143 +0000 UTC m=+1606.435916547" Oct 30 15:15:02.519152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:02.519109 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" podStartSLOduration=2.519096721 podStartE2EDuration="2.519096721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:02.515249484 +0000 UTC m=+1606.474290890" watchObservedRunningTime="2026-10-30 15:15:02.519096721 +0000 UTC m=+1606.478138128" Oct 30 15:15:03.028096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.028074 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:15:03.075109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.075087 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:15:03.108186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.108153 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume\") pod \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " Oct 30 15:15:03.108398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.108373 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn\" (UniqueName: \"kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn\") pod \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\" (UID: \"6c1aa13d-72fd-4e10-9733-4773fc7c69b5\") " Oct 30 15:15:03.108570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.108544 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume" pod "6c1aa13d-72fd-4e10-9733-4773fc7c69b5" (UID: "6c1aa13d-72fd-4e10-9733-4773fc7c69b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:03.109103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.109086 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:03.110499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.110477 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn" pod "6c1aa13d-72fd-4e10-9733-4773fc7c69b5" (UID: "6c1aa13d-72fd-4e10-9733-4773fc7c69b5"). InnerVolumeSpecName "kube-api-access-bjxbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:03.210534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.210461 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjxbn\" (UniqueName: \"kubernetes.io/projected/6c1aa13d-72fd-4e10-9733-4773fc7c69b5-kube-api-access-bjxbn\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:03.240464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.240431 6554 generic.go:356] "Generic (PLEG): container finished" podID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerID="a00be06f7b3e9b34b8e59ffd884f82d106ab139a3a48a27a9897e457217c43ef" exitCode=60 Oct 30 15:15:03.240633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.240512 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" event={"ID":"b8c26b7b-085b-424e-abe3-5111cb128b1a","Type":"ContainerDied","Data":"a00be06f7b3e9b34b8e59ffd884f82d106ab139a3a48a27a9897e457217c43ef"} Oct 30 15:15:03.241272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.240799 6554 scope.go:122] "RemoveContainer" containerID="a00be06f7b3e9b34b8e59ffd884f82d106ab139a3a48a27a9897e457217c43ef" Oct 30 15:15:03.242321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.242300 6554 generic.go:356] "Generic (PLEG): container finished" podID="98e02f5f-6d5a-4029-8323-f100e719c462" containerID="cd8ab3e006fe8ad3f4cee9a7daa658267b6535b8f7c96203adbce25b40bc8595" exitCode=60 Oct 30 15:15:03.242450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.242378 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerDied","Data":"cd8ab3e006fe8ad3f4cee9a7daa658267b6535b8f7c96203adbce25b40bc8595"} Oct 30 15:15:03.242690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.242663 6554 scope.go:122] "RemoveContainer" containerID="cd8ab3e006fe8ad3f4cee9a7daa658267b6535b8f7c96203adbce25b40bc8595" Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.246651 6554 generic.go:356] "Generic (PLEG): container finished" podID="640e515c-16b2-49d4-beea-43ee099f7f09" containerID="bf784696ace29def89d15c1eab0884ecde007a7646c4de20456c0217fc707290" exitCode=7 Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.246807 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerDied","Data":"bf784696ace29def89d15c1eab0884ecde007a7646c4de20456c0217fc707290"} Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.247240 6554 scope.go:122] "RemoveContainer" containerID="bf784696ace29def89d15c1eab0884ecde007a7646c4de20456c0217fc707290" Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.254882 6554 generic.go:356] "Generic (PLEG): container finished" podID="6c1aa13d-72fd-4e10-9733-4773fc7c69b5" containerID="be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f" exitCode=0 Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.255041 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.255258 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" event={"ID":"6c1aa13d-72fd-4e10-9733-4773fc7c69b5","Type":"ContainerDied","Data":"be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f"} Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.255320 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl" event={"ID":"6c1aa13d-72fd-4e10-9733-4773fc7c69b5","Type":"ContainerDied","Data":"0b8eb7a31aad4290b561f5591d62851fe82771bebc1a935924f7e2eed465b765"} Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.255603 6554 scope.go:122] "RemoveContainer" containerID="be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f" Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.257970 6554 generic.go:356] "Generic (PLEG): container finished" podID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerID="2c4e92b9152903f47a8ba7be54601eeb9cbf2d742f4faf0e363ec2e90a5f0e72" exitCode=60 Oct 30 15:15:03.258383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.258080 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerDied","Data":"2c4e92b9152903f47a8ba7be54601eeb9cbf2d742f4faf0e363ec2e90a5f0e72"} Oct 30 15:15:03.259783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.258474 6554 scope.go:122] "RemoveContainer" containerID="2c4e92b9152903f47a8ba7be54601eeb9cbf2d742f4faf0e363ec2e90a5f0e72" Oct 30 15:15:03.260408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.260382 6554 generic.go:356] "Generic (PLEG): container finished" podID="45e987c8-a676-4474-94ad-89b06c778051" containerID="46a017dbed3a705ad0380b20072f0d0187a32fed5deb9b39e84bd47709394f07" exitCode=60 Oct 30 15:15:03.260545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.260516 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerDied","Data":"46a017dbed3a705ad0380b20072f0d0187a32fed5deb9b39e84bd47709394f07"} Oct 30 15:15:03.261185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.261169 6554 scope.go:122] "RemoveContainer" containerID="46a017dbed3a705ad0380b20072f0d0187a32fed5deb9b39e84bd47709394f07" Oct 30 15:15:03.262800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.262780 6554 generic.go:356] "Generic (PLEG): container finished" podID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerID="be3e1ede7390366d91a709e02276ac20dc5cc75a3cfbaf2455bec5f3809eaf4a" exitCode=60 Oct 30 15:15:03.262892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.262862 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" event={"ID":"048b6a4e-fbee-489e-a415-698df68d4bb5","Type":"ContainerDied","Data":"be3e1ede7390366d91a709e02276ac20dc5cc75a3cfbaf2455bec5f3809eaf4a"} Oct 30 15:15:03.263236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.263213 6554 scope.go:122] "RemoveContainer" containerID="be3e1ede7390366d91a709e02276ac20dc5cc75a3cfbaf2455bec5f3809eaf4a" Oct 30 15:15:03.272723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.272701 6554 generic.go:356] "Generic (PLEG): container finished" podID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerID="4e73f42d072a86676fffdc31cf241bb2966de9884321a4c19bba9d916d49731a" exitCode=60 Oct 30 15:15:03.273011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.272990 6554 scope.go:122] "RemoveContainer" containerID="4e73f42d072a86676fffdc31cf241bb2966de9884321a4c19bba9d916d49731a" Oct 30 15:15:03.273574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.272856 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerDied","Data":"4e73f42d072a86676fffdc31cf241bb2966de9884321a4c19bba9d916d49731a"} Oct 30 15:15:03.276303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.276281 6554 generic.go:356] "Generic (PLEG): container finished" podID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerID="88e333f798570920db69d1932a8809348763f9af155b2394d6d2f991ca04cd2a" exitCode=60 Oct 30 15:15:03.276435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.276403 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerDied","Data":"88e333f798570920db69d1932a8809348763f9af155b2394d6d2f991ca04cd2a"} Oct 30 15:15:03.276725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.276708 6554 scope.go:122] "RemoveContainer" containerID="88e333f798570920db69d1932a8809348763f9af155b2394d6d2f991ca04cd2a" Oct 30 15:15:03.279828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.279807 6554 generic.go:356] "Generic (PLEG): container finished" podID="7751512c-6473-467a-8fad-4fb8bf588636" containerID="006d25cf782add05135911df608585d1440099907c2fc65d60f97a26ee6009ce" exitCode=60 Oct 30 15:15:03.279909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.279888 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerDied","Data":"006d25cf782add05135911df608585d1440099907c2fc65d60f97a26ee6009ce"} Oct 30 15:15:03.280199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.280180 6554 scope.go:122] "RemoveContainer" containerID="006d25cf782add05135911df608585d1440099907c2fc65d60f97a26ee6009ce" Oct 30 15:15:03.285806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.285786 6554 scope.go:122] "RemoveContainer" containerID="be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f" Oct 30 15:15:03.286173 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:03.286141 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f\": container with ID starting with be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f not found: ID does not exist" containerID="be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f" Oct 30 15:15:03.286267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.286181 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f"} err="failed to get container status \"be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f\": rpc error: code = NotFound desc = could not find container \"be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f\": container with ID starting with be3f9009383a4c1f572a1411c329825340b31b5be9391bb150e2e36713dcfb1f not found: ID does not exist" Oct 30 15:15:03.670120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.670087 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl"] Oct 30 15:15:03.674505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:03.674478 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-147bdb-b9b7dc975-h6gjl"] Oct 30 15:15:04.164041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.164009 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1aa13d-72fd-4e10-9733-4773fc7c69b5" path="/var/lib/kubelet/pods/6c1aa13d-72fd-4e10-9733-4773fc7c69b5/volumes" Oct 30 15:15:04.290665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.290640 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerStarted","Data":"9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc"} Oct 30 15:15:04.292831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.292805 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerStarted","Data":"6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab"} Oct 30 15:15:04.294693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.294670 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" event={"ID":"048b6a4e-fbee-489e-a415-698df68d4bb5","Type":"ContainerStarted","Data":"a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485"} Oct 30 15:15:04.296536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.296504 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerStarted","Data":"6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5"} Oct 30 15:15:04.298407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.298382 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerStarted","Data":"4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8"} Oct 30 15:15:04.300736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.300715 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerStarted","Data":"b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba"} Oct 30 15:15:04.302548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.302525 6554 generic.go:356] "Generic (PLEG): container finished" podID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerID="457e7b7934aa3cc2f45be768dcc7c32c918116cecf609e636ba1a54a45c09d16" exitCode=60 Oct 30 15:15:04.302634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.302599 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" event={"ID":"b8c26b7b-085b-424e-abe3-5111cb128b1a","Type":"ContainerDied","Data":"457e7b7934aa3cc2f45be768dcc7c32c918116cecf609e636ba1a54a45c09d16"} Oct 30 15:15:04.302685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.302646 6554 scope.go:122] "RemoveContainer" containerID="a00be06f7b3e9b34b8e59ffd884f82d106ab139a3a48a27a9897e457217c43ef" Oct 30 15:15:04.302867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.302849 6554 scope.go:122] "RemoveContainer" containerID="457e7b7934aa3cc2f45be768dcc7c32c918116cecf609e636ba1a54a45c09d16" Oct 30 15:15:04.303211 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:04.303182 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw_odh-ai-gateway-infra(b8c26b7b-085b-424e-abe3-5111cb128b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" Oct 30 15:15:04.304510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.304488 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerStarted","Data":"d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400"} Oct 30 15:15:04.306289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.306270 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerStarted","Data":"e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66"} Oct 30 15:15:04.498223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.492148 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" podStartSLOduration=4.492130749 podStartE2EDuration="4.492130749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:04.486168035 +0000 UTC m=+1608.445209440" watchObservedRunningTime="2026-10-30 15:15:04.492130749 +0000 UTC m=+1608.451172156" Oct 30 15:15:04.619943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:04.619865 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" podStartSLOduration=4.619851139 podStartE2EDuration="4.619851139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:04.61883149 +0000 UTC m=+1608.577872908" watchObservedRunningTime="2026-10-30 15:15:04.619851139 +0000 UTC m=+1608.578892543" Oct 30 15:15:05.348451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.348323 6554 generic.go:356] "Generic (PLEG): container finished" podID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerID="4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8" exitCode=60 Oct 30 15:15:05.350391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.350238 6554 scope.go:122] "RemoveContainer" containerID="4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8" Oct 30 15:15:05.351281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.350650 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerDied","Data":"4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8"} Oct 30 15:15:05.351281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.350771 6554 scope.go:122] "RemoveContainer" containerID="88e333f798570920db69d1932a8809348763f9af155b2394d6d2f991ca04cd2a" Oct 30 15:15:05.351281 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.350795 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c_odh-ai-gateway-infra(1d0f5c6b-6606-427b-b356-2275618e69ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" Oct 30 15:15:05.358719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.358691 6554 generic.go:356] "Generic (PLEG): container finished" podID="7751512c-6473-467a-8fad-4fb8bf588636" containerID="b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba" exitCode=60 Oct 30 15:15:05.358933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.358900 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerDied","Data":"b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba"} Oct 30 15:15:05.361532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.360539 6554 scope.go:122] "RemoveContainer" containerID="b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba" Oct 30 15:15:05.362104 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.362075 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22_odh-ai-gateway-infra(7751512c-6473-467a-8fad-4fb8bf588636)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" podUID="7751512c-6473-467a-8fad-4fb8bf588636" Oct 30 15:15:05.374626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.374581 6554 scope.go:122] "RemoveContainer" containerID="457e7b7934aa3cc2f45be768dcc7c32c918116cecf609e636ba1a54a45c09d16" Oct 30 15:15:05.375052 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.375026 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw_odh-ai-gateway-infra(b8c26b7b-085b-424e-abe3-5111cb128b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" Oct 30 15:15:05.385429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.381893 6554 generic.go:356] "Generic (PLEG): container finished" podID="98e02f5f-6d5a-4029-8323-f100e719c462" containerID="d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400" exitCode=60 Oct 30 15:15:05.385429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.381966 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerDied","Data":"d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400"} Oct 30 15:15:05.385429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.384892 6554 scope.go:122] "RemoveContainer" containerID="d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400" Oct 30 15:15:05.385429 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.385338 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql_odh-ai-gateway-infra(98e02f5f-6d5a-4029-8323-f100e719c462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" Oct 30 15:15:05.391540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.389571 6554 generic.go:356] "Generic (PLEG): container finished" podID="640e515c-16b2-49d4-beea-43ee099f7f09" containerID="e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66" exitCode=7 Oct 30 15:15:05.391540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.389687 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerDied","Data":"e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66"} Oct 30 15:15:05.391540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.390303 6554 scope.go:122] "RemoveContainer" containerID="e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66" Oct 30 15:15:05.392108 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.392084 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp_odh-ai-gateway-infra(640e515c-16b2-49d4-beea-43ee099f7f09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" Oct 30 15:15:05.392545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.392518 6554 scope.go:122] "RemoveContainer" containerID="006d25cf782add05135911df608585d1440099907c2fc65d60f97a26ee6009ce" Oct 30 15:15:05.395704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.395674 6554 generic.go:356] "Generic (PLEG): container finished" podID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerID="9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc" exitCode=60 Oct 30 15:15:05.395809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.395757 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerDied","Data":"9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc"} Oct 30 15:15:05.396128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.396110 6554 scope.go:122] "RemoveContainer" containerID="9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc" Oct 30 15:15:05.396436 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.396411 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss_odh-ai-gateway-infra(d799f44c-675e-440f-9c81-8ee0583ca6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" Oct 30 15:15:05.398485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.398463 6554 generic.go:356] "Generic (PLEG): container finished" podID="45e987c8-a676-4474-94ad-89b06c778051" containerID="6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab" exitCode=60 Oct 30 15:15:05.398839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.398784 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerDied","Data":"6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab"} Oct 30 15:15:05.399743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.399456 6554 scope.go:122] "RemoveContainer" containerID="6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab" Oct 30 15:15:05.399873 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.399792 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt_odh-ai-gateway-infra(45e987c8-a676-4474-94ad-89b06c778051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" podUID="45e987c8-a676-4474-94ad-89b06c778051" Oct 30 15:15:05.411469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.411442 6554 generic.go:356] "Generic (PLEG): container finished" podID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerID="a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485" exitCode=60 Oct 30 15:15:05.411604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.411576 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" event={"ID":"048b6a4e-fbee-489e-a415-698df68d4bb5","Type":"ContainerDied","Data":"a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485"} Oct 30 15:15:05.411822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.411804 6554 scope.go:122] "RemoveContainer" containerID="a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485" Oct 30 15:15:05.412159 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.412138 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b_odh-ai-gateway-infra(048b6a4e-fbee-489e-a415-698df68d4bb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" Oct 30 15:15:05.416670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.416645 6554 generic.go:356] "Generic (PLEG): container finished" podID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerID="6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5" exitCode=60 Oct 30 15:15:05.416848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.416708 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerDied","Data":"6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5"} Oct 30 15:15:05.417485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.417117 6554 scope.go:122] "RemoveContainer" containerID="6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5" Oct 30 15:15:05.417628 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:05.417603 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj_odh-ai-gateway-infra(a7edcdb2-552f-4a1c-b36f-dfee06683616)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" Oct 30 15:15:05.429004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.428963 6554 scope.go:122] "RemoveContainer" containerID="cd8ab3e006fe8ad3f4cee9a7daa658267b6535b8f7c96203adbce25b40bc8595" Oct 30 15:15:05.446220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.446199 6554 scope.go:122] "RemoveContainer" containerID="bf784696ace29def89d15c1eab0884ecde007a7646c4de20456c0217fc707290" Oct 30 15:15:05.464439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.464413 6554 scope.go:122] "RemoveContainer" containerID="2c4e92b9152903f47a8ba7be54601eeb9cbf2d742f4faf0e363ec2e90a5f0e72" Oct 30 15:15:05.478493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.478474 6554 scope.go:122] "RemoveContainer" containerID="46a017dbed3a705ad0380b20072f0d0187a32fed5deb9b39e84bd47709394f07" Oct 30 15:15:05.491840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.491823 6554 scope.go:122] "RemoveContainer" containerID="be3e1ede7390366d91a709e02276ac20dc5cc75a3cfbaf2455bec5f3809eaf4a" Oct 30 15:15:05.505428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:05.505408 6554 scope.go:122] "RemoveContainer" containerID="4e73f42d072a86676fffdc31cf241bb2966de9884321a4c19bba9d916d49731a" Oct 30 15:15:06.427245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.427212 6554 scope.go:122] "RemoveContainer" containerID="a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485" Oct 30 15:15:06.427721 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.427573 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b_odh-ai-gateway-infra(048b6a4e-fbee-489e-a415-698df68d4bb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" Oct 30 15:15:06.429236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.429210 6554 scope.go:122] "RemoveContainer" containerID="6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5" Oct 30 15:15:06.429543 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.429514 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj_odh-ai-gateway-infra(a7edcdb2-552f-4a1c-b36f-dfee06683616)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" Oct 30 15:15:06.431163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.431135 6554 scope.go:122] "RemoveContainer" containerID="4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8" Oct 30 15:15:06.431454 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.431429 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c_odh-ai-gateway-infra(1d0f5c6b-6606-427b-b356-2275618e69ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" Oct 30 15:15:06.432939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.432920 6554 scope.go:122] "RemoveContainer" containerID="b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba" Oct 30 15:15:06.433182 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.433155 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22_odh-ai-gateway-infra(7751512c-6473-467a-8fad-4fb8bf588636)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" podUID="7751512c-6473-467a-8fad-4fb8bf588636" Oct 30 15:15:06.434693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.434677 6554 scope.go:122] "RemoveContainer" containerID="d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400" Oct 30 15:15:06.434916 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.434894 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql_odh-ai-gateway-infra(98e02f5f-6d5a-4029-8323-f100e719c462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" Oct 30 15:15:06.436193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.436176 6554 scope.go:122] "RemoveContainer" containerID="e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66" Oct 30 15:15:06.436429 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.436412 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp_odh-ai-gateway-infra(640e515c-16b2-49d4-beea-43ee099f7f09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" Oct 30 15:15:06.437794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.437775 6554 scope.go:122] "RemoveContainer" containerID="9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc" Oct 30 15:15:06.438040 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.438014 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss_odh-ai-gateway-infra(d799f44c-675e-440f-9c81-8ee0583ca6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" Oct 30 15:15:06.439422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:06.439402 6554 scope.go:122] "RemoveContainer" containerID="6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab" Oct 30 15:15:06.439638 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:06.439619 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt_odh-ai-gateway-infra(45e987c8-a676-4474-94ad-89b06c778051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" podUID="45e987c8-a676-4474-94ad-89b06c778051" Oct 30 15:15:09.244139 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:09.244099 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:15:15.147610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.147580 6554 scope.go:122] "RemoveContainer" containerID="a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485" Oct 30 15:15:15.299492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.299455 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr"] Oct 30 15:15:15.299817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.299777 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" containerID="cri-o://e526ad9992c1e05a1d7fd4f5ef53735f7e0c983885737e00588bc08cd10f531f" gracePeriod=30 Oct 30 15:15:15.314309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.314272 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f"] Oct 30 15:15:15.314543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.314517 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" containerID="cri-o://f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a" gracePeriod=30 Oct 30 15:15:15.331600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.331568 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255"] Oct 30 15:15:15.331841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.331813 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" containerID="cri-o://8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506" gracePeriod=30 Oct 30 15:15:15.337073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.337048 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7"] Oct 30 15:15:15.337261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:15.337238 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" containerID="cri-o://437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c" gracePeriod=30 Oct 30 15:15:16.302269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.301941 6554 scope.go:122] "RemoveContainer" containerID="d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400" Oct 30 15:15:16.644861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.642862 6554 generic.go:356] "Generic (PLEG): container finished" podID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerID="673fa8e86dd1bd41b42e7cad4b7b6a363b95821180fc0101713e9aac785cd901" exitCode=60 Oct 30 15:15:16.644861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.643185 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" event={"ID":"048b6a4e-fbee-489e-a415-698df68d4bb5","Type":"ContainerDied","Data":"673fa8e86dd1bd41b42e7cad4b7b6a363b95821180fc0101713e9aac785cd901"} Oct 30 15:15:16.644861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.643241 6554 scope.go:122] "RemoveContainer" containerID="a603d963d02c237a3c4b0f8fe9422453168cde69dba0a59d0126e93b4ced8485" Oct 30 15:15:16.644861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.643829 6554 scope.go:122] "RemoveContainer" containerID="673fa8e86dd1bd41b42e7cad4b7b6a363b95821180fc0101713e9aac785cd901" Oct 30 15:15:16.644861 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:16.644271 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b_odh-ai-gateway-infra(048b6a4e-fbee-489e-a415-698df68d4bb5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" Oct 30 15:15:16.752174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.751614 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c"] Oct 30 15:15:16.753954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.753439 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6c1aa13d-72fd-4e10-9733-4773fc7c69b5" containerName="payload-pre-processing" Oct 30 15:15:16.753954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.753464 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1aa13d-72fd-4e10-9733-4773fc7c69b5" containerName="payload-pre-processing" Oct 30 15:15:16.753954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.753761 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1aa13d-72fd-4e10-9733-4773fc7c69b5" containerName="payload-pre-processing" Oct 30 15:15:16.763369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.760741 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:16.783375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.782801 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c"] Oct 30 15:15:16.867606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.867569 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:15:16.925046 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.925013 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:16.925230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.925208 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqnc4\" (UniqueName: \"kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:16.925392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:16.925376 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.027255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.027225 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.027387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.027375 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqnc4\" (UniqueName: \"kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.027488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.027476 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.027888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.027868 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.029677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.029659 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.052356 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.052333 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqnc4\" (UniqueName: \"kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4\") pod \"maas-api-revoke-keys-e2e-api-a-1b7814-wf87c\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.132206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.132130 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:17.147537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.147516 6554 scope.go:122] "RemoveContainer" containerID="b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba" Oct 30 15:15:17.147684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.147666 6554 scope.go:122] "RemoveContainer" containerID="6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab" Oct 30 15:15:17.527674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.527627 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c"] Oct 30 15:15:17.671766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.671732 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" event={"ID":"e8322d9f-3fe5-478b-a966-aa713bdc06e2","Type":"ContainerStarted","Data":"545f3727f188017db37d81472380a624aa17c51b927f806276e827c04299d450"} Oct 30 15:15:17.688137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.688107 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerStarted","Data":"d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99"} Oct 30 15:15:17.714624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:17.714597 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b"] Oct 30 15:15:18.003045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.003019 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:18.181686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.181649 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr\" (UniqueName: \"kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr\") pod \"048b6a4e-fbee-489e-a415-698df68d4bb5\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " Oct 30 15:15:18.181923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.181738 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca\") pod \"048b6a4e-fbee-489e-a415-698df68d4bb5\" (UID: \"048b6a4e-fbee-489e-a415-698df68d4bb5\") " Oct 30 15:15:18.183639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.183082 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca" pod "048b6a4e-fbee-489e-a415-698df68d4bb5" (UID: "048b6a4e-fbee-489e-a415-698df68d4bb5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:18.185020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.185002 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr" pod "048b6a4e-fbee-489e-a415-698df68d4bb5" (UID: "048b6a4e-fbee-489e-a415-698df68d4bb5"). InnerVolumeSpecName "kube-api-access-t9vgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:18.187167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.187146 6554 scope.go:122] "RemoveContainer" containerID="6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5" Oct 30 15:15:18.188230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.187952 6554 scope.go:122] "RemoveContainer" containerID="457e7b7934aa3cc2f45be768dcc7c32c918116cecf609e636ba1a54a45c09d16" Oct 30 15:15:18.188439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.188419 6554 scope.go:122] "RemoveContainer" containerID="4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8" Oct 30 15:15:18.188643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.188559 6554 scope.go:122] "RemoveContainer" containerID="9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc" Oct 30 15:15:18.285148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.285074 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9vgr\" (UniqueName: \"kubernetes.io/projected/048b6a4e-fbee-489e-a415-698df68d4bb5-kube-api-access-t9vgr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:18.285148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.285101 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/048b6a4e-fbee-489e-a415-698df68d4bb5-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:18.700562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.700527 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" event={"ID":"e8322d9f-3fe5-478b-a966-aa713bdc06e2","Type":"ContainerStarted","Data":"c717aa023fc0eb84999982a2a2b0eb004a30b46e3312828218a4b5b86a579090"} Oct 30 15:15:18.702495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.702467 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerStarted","Data":"75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a"} Oct 30 15:15:18.704455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.704424 6554 generic.go:356] "Generic (PLEG): container finished" podID="7751512c-6473-467a-8fad-4fb8bf588636" containerID="767af4a031b5dad5b02c47d89219724510af0d73aa7c53c900282d01ef4d7f29" exitCode=60 Oct 30 15:15:18.704560 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.704497 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerDied","Data":"767af4a031b5dad5b02c47d89219724510af0d73aa7c53c900282d01ef4d7f29"} Oct 30 15:15:18.704560 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.704549 6554 scope.go:122] "RemoveContainer" containerID="b102623f54d9007121be5c1b375dc58b2fd0bc0a97e28dcce0ff99a1afe515ba" Oct 30 15:15:18.705388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.704934 6554 scope.go:122] "RemoveContainer" containerID="767af4a031b5dad5b02c47d89219724510af0d73aa7c53c900282d01ef4d7f29" Oct 30 15:15:18.705388 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:18.705241 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22_odh-ai-gateway-infra(7751512c-6473-467a-8fad-4fb8bf588636)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" podUID="7751512c-6473-467a-8fad-4fb8bf588636" Oct 30 15:15:18.709402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.709376 6554 generic.go:356] "Generic (PLEG): container finished" podID="98e02f5f-6d5a-4029-8323-f100e719c462" containerID="d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99" exitCode=60 Oct 30 15:15:18.709498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.709441 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerDied","Data":"d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99"} Oct 30 15:15:18.709760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.709745 6554 scope.go:122] "RemoveContainer" containerID="d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99" Oct 30 15:15:18.709983 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:18.709959 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql_odh-ai-gateway-infra(98e02f5f-6d5a-4029-8323-f100e719c462)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" Oct 30 15:15:18.711487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.711458 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerStarted","Data":"7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd"} Oct 30 15:15:18.713398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.713379 6554 generic.go:356] "Generic (PLEG): container finished" podID="45e987c8-a676-4474-94ad-89b06c778051" containerID="fa29a909c5f4db018b1fcd9f48c24666932fa2cee8955eeb084dd4168f4ea435" exitCode=60 Oct 30 15:15:18.713477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.713433 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerDied","Data":"fa29a909c5f4db018b1fcd9f48c24666932fa2cee8955eeb084dd4168f4ea435"} Oct 30 15:15:18.713797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.713780 6554 scope.go:122] "RemoveContainer" containerID="fa29a909c5f4db018b1fcd9f48c24666932fa2cee8955eeb084dd4168f4ea435" Oct 30 15:15:18.714106 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:18.714076 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt_odh-ai-gateway-infra(45e987c8-a676-4474-94ad-89b06c778051)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" podUID="45e987c8-a676-4474-94ad-89b06c778051" Oct 30 15:15:18.714816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.714790 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" Oct 30 15:15:18.714915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.714820 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b" event={"ID":"048b6a4e-fbee-489e-a415-698df68d4bb5","Type":"ContainerDied","Data":"066c145b1567e7ded34bffb66dd62ee039cbed3301aeb1d90f8f36c5f078dd0d"} Oct 30 15:15:18.724447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.724420 6554 scope.go:122] "RemoveContainer" containerID="d023035a9781d02ecb4dc5f5bd54479e8485dc0b4a9880106a4f0a625312a400" Oct 30 15:15:18.748736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.745769 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" podStartSLOduration=2.7457316240000003 podStartE2EDuration="2.745731624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:18.744270296 +0000 UTC m=+1622.703311700" watchObservedRunningTime="2026-10-30 15:15:18.745731624 +0000 UTC m=+1622.704773030" Oct 30 15:15:18.750508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.750309 6554 scope.go:122] "RemoveContainer" containerID="6e6c211a529f18ba8fb3ecb4cd0bc61e36cbc21650d4c70d207eb816b706b8ab" Oct 30 15:15:18.790920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.790854 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql"] Oct 30 15:15:18.800751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.800728 6554 scope.go:122] "RemoveContainer" containerID="673fa8e86dd1bd41b42e7cad4b7b6a363b95821180fc0101713e9aac785cd901" Oct 30 15:15:18.938688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.938655 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b"] Oct 30 15:15:18.979389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:18.973328 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-751ab3-29889555-vnh5b"] Oct 30 15:15:19.146980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.146953 6554 scope.go:122] "RemoveContainer" containerID="e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66" Oct 30 15:15:19.728194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.728107 6554 generic.go:356] "Generic (PLEG): container finished" podID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerID="75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a" exitCode=60 Oct 30 15:15:19.728194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.728131 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerDied","Data":"75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a"} Oct 30 15:15:19.728194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.728190 6554 scope.go:122] "RemoveContainer" containerID="4864826a2b635b5aac2909f1cb8a4f1d98f4f84033f1ffacb6a2b6c12b186db8" Oct 30 15:15:19.728732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.728549 6554 scope.go:122] "RemoveContainer" containerID="75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a" Oct 30 15:15:19.728885 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:19.728864 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c_odh-ai-gateway-infra(1d0f5c6b-6606-427b-b356-2275618e69ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" Oct 30 15:15:19.732463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.732437 6554 generic.go:356] "Generic (PLEG): container finished" podID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerID="e90506d84ac4238cdb451355b1c293418b795744b35100ccad8c93ca4a2c9fc2" exitCode=60 Oct 30 15:15:19.732599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.732510 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" event={"ID":"b8c26b7b-085b-424e-abe3-5111cb128b1a","Type":"ContainerDied","Data":"e90506d84ac4238cdb451355b1c293418b795744b35100ccad8c93ca4a2c9fc2"} Oct 30 15:15:19.732923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.732894 6554 scope.go:122] "RemoveContainer" containerID="e90506d84ac4238cdb451355b1c293418b795744b35100ccad8c93ca4a2c9fc2" Oct 30 15:15:19.733274 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:19.733251 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw_odh-ai-gateway-infra(b8c26b7b-085b-424e-abe3-5111cb128b1a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" Oct 30 15:15:19.736486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.736466 6554 generic.go:356] "Generic (PLEG): container finished" podID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerID="7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd" exitCode=60 Oct 30 15:15:19.736583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.736529 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerDied","Data":"7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd"} Oct 30 15:15:19.736978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.736949 6554 scope.go:122] "RemoveContainer" containerID="7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd" Oct 30 15:15:19.737294 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:19.737276 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss_odh-ai-gateway-infra(d799f44c-675e-440f-9c81-8ee0583ca6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" Oct 30 15:15:19.741857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.741828 6554 generic.go:356] "Generic (PLEG): container finished" podID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerID="03878a41d235f6b9cb657f0b9541f260170a914e4a400b282773d81fd226f8dc" exitCode=60 Oct 30 15:15:19.741959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.741875 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerDied","Data":"03878a41d235f6b9cb657f0b9541f260170a914e4a400b282773d81fd226f8dc"} Oct 30 15:15:19.742297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.742278 6554 scope.go:122] "RemoveContainer" containerID="03878a41d235f6b9cb657f0b9541f260170a914e4a400b282773d81fd226f8dc" Oct 30 15:15:19.742698 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:19.742670 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj_odh-ai-gateway-infra(a7edcdb2-552f-4a1c-b36f-dfee06683616)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" Oct 30 15:15:19.743891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.743845 6554 generic.go:356] "Generic (PLEG): container finished" podID="e8322d9f-3fe5-478b-a966-aa713bdc06e2" containerID="c717aa023fc0eb84999982a2a2b0eb004a30b46e3312828218a4b5b86a579090" exitCode=0 Oct 30 15:15:19.743891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.743881 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" event={"ID":"e8322d9f-3fe5-478b-a966-aa713bdc06e2","Type":"ContainerDied","Data":"c717aa023fc0eb84999982a2a2b0eb004a30b46e3312828218a4b5b86a579090"} Oct 30 15:15:19.744641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.744621 6554 scope.go:122] "RemoveContainer" containerID="457e7b7934aa3cc2f45be768dcc7c32c918116cecf609e636ba1a54a45c09d16" Oct 30 15:15:19.797783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.797743 6554 scope.go:122] "RemoveContainer" containerID="9c3b01703a2b6f4fb46e4ba87b355bd3f75793c0785449ae8ab43c053dcf2fbc" Oct 30 15:15:19.820454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.820312 6554 scope.go:122] "RemoveContainer" containerID="6f09133a02d88515046a55a491addb3c45f980111402ae35b122e093a74aa4d5" Oct 30 15:15:19.889736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.889638 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss"] Oct 30 15:15:19.904263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.903977 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22"] Oct 30 15:15:19.996689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:19.996628 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:20.019706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.019671 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca\") pod \"98e02f5f-6d5a-4029-8323-f100e719c462\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " Oct 30 15:15:20.019976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.019953 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55\" (UniqueName: \"kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55\") pod \"98e02f5f-6d5a-4029-8323-f100e719c462\" (UID: \"98e02f5f-6d5a-4029-8323-f100e719c462\") " Oct 30 15:15:20.020549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.020523 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca" pod "98e02f5f-6d5a-4029-8323-f100e719c462" (UID: "98e02f5f-6d5a-4029-8323-f100e719c462"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.021609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.021572 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98e02f5f-6d5a-4029-8323-f100e719c462-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.022636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.022612 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55" pod "98e02f5f-6d5a-4029-8323-f100e719c462" (UID: "98e02f5f-6d5a-4029-8323-f100e719c462"). InnerVolumeSpecName "kube-api-access-2tw55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.049043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.049013 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c"] Oct 30 15:15:20.113373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.113325 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:20.123481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.123442 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca\") pod \"7751512c-6473-467a-8fad-4fb8bf588636\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " Oct 30 15:15:20.123728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.123701 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq\" (UniqueName: \"kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq\") pod \"7751512c-6473-467a-8fad-4fb8bf588636\" (UID: \"7751512c-6473-467a-8fad-4fb8bf588636\") " Oct 30 15:15:20.123839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.123770 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca" pod "7751512c-6473-467a-8fad-4fb8bf588636" (UID: "7751512c-6473-467a-8fad-4fb8bf588636"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.124768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.124736 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7751512c-6473-467a-8fad-4fb8bf588636-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.124768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.124770 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tw55\" (UniqueName: \"kubernetes.io/projected/98e02f5f-6d5a-4029-8323-f100e719c462-kube-api-access-2tw55\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.126081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.126053 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq" pod "7751512c-6473-467a-8fad-4fb8bf588636" (UID: "7751512c-6473-467a-8fad-4fb8bf588636"). InnerVolumeSpecName "kube-api-access-nhsjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.140212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.140187 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt"] Oct 30 15:15:20.182389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.168425 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" path="/var/lib/kubelet/pods/048b6a4e-fbee-489e-a415-698df68d4bb5/volumes" Oct 30 15:15:20.227422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.227393 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhsjq\" (UniqueName: \"kubernetes.io/projected/7751512c-6473-467a-8fad-4fb8bf588636-kube-api-access-nhsjq\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.426372 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:20.426304 6554 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7edcdb2_552f_4a1c_b36f_dfee06683616.slice/crio-03878a41d235f6b9cb657f0b9541f260170a914e4a400b282773d81fd226f8dc.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7edcdb2_552f_4a1c_b36f_dfee06683616.slice/crio-03878a41d235f6b9cb657f0b9541f260170a914e4a400b282773d81fd226f8dc.scope: no such file or directory Oct 30 15:15:20.426372 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:20.426368 6554 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0f5c6b_6606_427b_b356_2275618e69ea.slice/crio-75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0f5c6b_6606_427b_b356_2275618e69ea.slice/crio-75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a.scope: no such file or directory Oct 30 15:15:20.426557 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:20.426397 6554 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd799f44c_675e_440f_9c81_8ee0583ca6cd.slice/crio-7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd799f44c_675e_440f_9c81_8ee0583ca6cd.slice/crio-7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd.scope: no such file or directory Oct 30 15:15:20.426557 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:20.426417 6554 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c26b7b_085b_424e_abe3_5111cb128b1a.slice/crio-e90506d84ac4238cdb451355b1c293418b795744b35100ccad8c93ca4a2c9fc2.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c26b7b_085b_424e_abe3_5111cb128b1a.slice/crio-e90506d84ac4238cdb451355b1c293418b795744b35100ccad8c93ca4a2c9fc2.scope: no such file or directory Oct 30 15:15:20.517410 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:20.516145 6554 watcher.go:93] Error while processing event ("/sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640e515c_16b2_49d4_beea_43ee099f7f09.slice/crio-cae1d32fd3069c8e1df945b3fcfe3b435956d7217703224b9d83c52169541a98.scope": 0x40000100 == IN_CREATE|IN_ISDIR): inotify_add_watch /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640e515c_16b2_49d4_beea_43ee099f7f09.slice/crio-cae1d32fd3069c8e1df945b3fcfe3b435956d7217703224b9d83c52169541a98.scope: no such file or directory Oct 30 15:15:20.553402 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:20.552723 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-conmon-437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6d4be6_01e2_4bcb_ad1e_8c5fba1f6f6c.slice/crio-conmon-8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice/crio-1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351672e0_ce2f_4ea0_8987_0b2471da2f6b.slice/crio-conmon-f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048b6a4e_fbee_489e_a415_698df68d4bb5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:20.584375 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:20.580752 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-conmon-437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6d4be6_01e2_4bcb_ad1e_8c5fba1f6f6c.slice/crio-8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351672e0_ce2f_4ea0_8987_0b2471da2f6b.slice/crio-conmon-f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351672e0_ce2f_4ea0_8987_0b2471da2f6b.slice/crio-f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice/crio-1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:20.587545 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:20.584744 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-conmon-437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:20.592535 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:20.586523 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6d4be6_01e2_4bcb_ad1e_8c5fba1f6f6c.slice/crio-8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6d4be6_01e2_4bcb_ad1e_8c5fba1f6f6c.slice/crio-conmon-8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-767af4a031b5dad5b02c47d89219724510af0d73aa7c53c900282d01ef4d7f29.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048b6a4e_fbee_489e_a415_698df68d4bb5.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-conmon-767af4a031b5dad5b02c47d89219724510af0d73aa7c53c900282d01ef4d7f29.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e987c8_a676_4474_94ad_89b06c778051.slice/crio-fa29a909c5f4db018b1fcd9f48c24666932fa2cee8955eeb084dd4168f4ea435.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45e987c8_a676_4474_94ad_89b06c778051.slice/crio-conmon-fa29a909c5f4db018b1fcd9f48c24666932fa2cee8955eeb084dd4168f4ea435.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c54aae_4a98_45eb_be48_ff6a4c21d35f.slice/crio-conmon-437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice/crio-1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351672e0_ce2f_4ea0_8987_0b2471da2f6b.slice/crio-f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice/crio-d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02f5f_6d5a_4029_8323_f100e719c462.slice/crio-conmon-d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:20.697035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.696829 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:20.792574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.792493 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p\" (UniqueName: \"kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p\") pod \"45e987c8-a676-4474-94ad-89b06c778051\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " Oct 30 15:15:20.793270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.793238 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca\") pod \"45e987c8-a676-4474-94ad-89b06c778051\" (UID: \"45e987c8-a676-4474-94ad-89b06c778051\") " Oct 30 15:15:20.795175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.795139 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca" pod "45e987c8-a676-4474-94ad-89b06c778051" (UID: "45e987c8-a676-4474-94ad-89b06c778051"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.803543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.803503 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p" pod "45e987c8-a676-4474-94ad-89b06c778051" (UID: "45e987c8-a676-4474-94ad-89b06c778051"). InnerVolumeSpecName "kube-api-access-mzj5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.811761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.811732 6554 generic.go:356] "Generic (PLEG): container finished" podID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerID="8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506" exitCode=0 Oct 30 15:15:20.813186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.811910 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" event={"ID":"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c","Type":"ContainerDied","Data":"8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506"} Oct 30 15:15:20.816329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.816195 6554 generic.go:356] "Generic (PLEG): container finished" podID="f6b96720-1681-466e-b94d-07bc26a00523" containerID="e526ad9992c1e05a1d7fd4f5ef53735f7e0c983885737e00588bc08cd10f531f" exitCode=0 Oct 30 15:15:20.816329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.816279 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" event={"ID":"f6b96720-1681-466e-b94d-07bc26a00523","Type":"ContainerDied","Data":"e526ad9992c1e05a1d7fd4f5ef53735f7e0c983885737e00588bc08cd10f531f"} Oct 30 15:15:20.816916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.816811 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:15:20.820799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.820699 6554 generic.go:356] "Generic (PLEG): container finished" podID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerID="437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c" exitCode=0 Oct 30 15:15:20.820799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.820753 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" event={"ID":"69c54aae-4a98-45eb-be48-ff6a4c21d35f","Type":"ContainerDied","Data":"437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c"} Oct 30 15:15:20.822509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.822388 6554 generic.go:356] "Generic (PLEG): container finished" podID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerID="f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a" exitCode=0 Oct 30 15:15:20.822509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.822460 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" event={"ID":"351672e0-ce2f-4ea0-8987-0b2471da2f6b","Type":"ContainerDied","Data":"f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a"} Oct 30 15:15:20.825632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.823809 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" event={"ID":"7751512c-6473-467a-8fad-4fb8bf588636","Type":"ContainerDied","Data":"10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568"} Oct 30 15:15:20.825632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.823848 6554 scope.go:122] "RemoveContainer" containerID="767af4a031b5dad5b02c47d89219724510af0d73aa7c53c900282d01ef4d7f29" Oct 30 15:15:20.825632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.823946 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22" Oct 30 15:15:20.829251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.828425 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" event={"ID":"98e02f5f-6d5a-4029-8323-f100e719c462","Type":"ContainerDied","Data":"1b64392a348c451e194af9bf39d9e2f2e1fdb13173aa3a0ef5600585b42982af"} Oct 30 15:15:20.829251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.828450 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql" Oct 30 15:15:20.834103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.834080 6554 generic.go:356] "Generic (PLEG): container finished" podID="640e515c-16b2-49d4-beea-43ee099f7f09" containerID="cae1d32fd3069c8e1df945b3fcfe3b435956d7217703224b9d83c52169541a98" exitCode=60 Oct 30 15:15:20.834190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.834174 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerDied","Data":"cae1d32fd3069c8e1df945b3fcfe3b435956d7217703224b9d83c52169541a98"} Oct 30 15:15:20.836523 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.836506 6554 scope.go:122] "RemoveContainer" containerID="cae1d32fd3069c8e1df945b3fcfe3b435956d7217703224b9d83c52169541a98" Oct 30 15:15:20.836882 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:20.836865 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp_odh-ai-gateway-infra(640e515c-16b2-49d4-beea-43ee099f7f09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" Oct 30 15:15:20.844209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.839294 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" Oct 30 15:15:20.844426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.844320 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt" event={"ID":"45e987c8-a676-4474-94ad-89b06c778051","Type":"ContainerDied","Data":"c64da047df038dfb3f2eaeb1067c27cf9087e3b7a246d7f920f0b2e3f4461130"} Oct 30 15:15:20.846730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.846471 6554 scope.go:122] "RemoveContainer" containerID="d631b872b2ed3fd9fe1a5d548574d8141c0c54869013457c2331665b15bf9b99" Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.895107 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.895186 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.896681 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.896746 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.896857 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.896891 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.896918 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.896966 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.897017 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo\") pod \"f6b96720-1681-466e-b94d-07bc26a00523\" (UID: \"f6b96720-1681-466e-b94d-07bc26a00523\") " Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.897939 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e987c8-a676-4474-94ad-89b06c778051-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.910008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.897962 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzj5p\" (UniqueName: \"kubernetes.io/projected/45e987c8-a676-4474-94ad-89b06c778051-kube-api-access-mzj5p\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:20.911908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.911887 6554 scope.go:122] "RemoveContainer" containerID="e307e7accf102dd04f68befa17acaca07f6e43b0ab97cbab846b473367cece66" Oct 30 15:15:20.912570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.912229 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:20.915788 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.915513 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:20.919219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.919004 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:20.920128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.920100 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:20.923649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.922230 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:20.951231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.951196 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:20.951725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.951696 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "kube-api-access-92hcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:20.952399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.952324 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:20.997232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.996451 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql"] Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999723 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999746 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6b96720-1681-466e-b94d-07bc26a00523-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999759 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999774 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999797 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999810 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6b96720-1681-466e-b94d-07bc26a00523-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999822 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6b96720-1681-466e-b94d-07bc26a00523-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:20.999835 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92hcx\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-kube-api-access-92hcx\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.000863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.000099 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token" pod "f6b96720-1681-466e-b94d-07bc26a00523" (UID: "f6b96720-1681-466e-b94d-07bc26a00523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.002557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.002307 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-c69f40-29889555-x6qql"] Oct 30 15:15:21.002557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.002407 6554 scope.go:122] "RemoveContainer" containerID="fa29a909c5f4db018b1fcd9f48c24666932fa2cee8955eeb084dd4168f4ea435" Oct 30 15:15:21.028577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.028122 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:15:21.089437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.087881 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj"] Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.094636 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw"] Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.100851 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.100895 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101095 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101169 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101264 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101301 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101342 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101468 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.101527 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv\") pod \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\" (UID: \"351672e0-ce2f-4ea0-8987-0b2471da2f6b\") " Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.102988 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.103283 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.104607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.104101 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.105489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.105406 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.105592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.105564 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.106253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.106150 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.106253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.106172 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.106253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.106194 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6b96720-1681-466e-b94d-07bc26a00523-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.106253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.106207 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.106253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.106221 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.106253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.106235 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.114597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.114575 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "kube-api-access-5mzsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.114837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.114811 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.122965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.118969 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.122965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.120226 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt"] Oct 30 15:15:21.130657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.127702 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token" pod "351672e0-ce2f-4ea0-8987-0b2471da2f6b" (UID: "351672e0-ce2f-4ea0-8987-0b2471da2f6b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.142746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.142716 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-609709-29889555-h4wjt"] Oct 30 15:15:21.199806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.199764 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:15:21.203555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.203327 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:15:21.222020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.221728 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.222020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.221758 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.222020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.221772 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mzsv\" (UniqueName: \"kubernetes.io/projected/351672e0-ce2f-4ea0-8987-0b2471da2f6b-kube-api-access-5mzsv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.222020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.221787 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/351672e0-ce2f-4ea0-8987-0b2471da2f6b-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.232633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.232607 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22"] Oct 30 15:15:21.237796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.237776 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889555-94l22"] Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323278 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323382 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323469 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323507 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323549 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323582 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323649 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323680 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323722 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323759 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323797 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323877 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323917 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.323966 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.324068 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token\") pod \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\" (UID: \"69c54aae-4a98-45eb-be48-ff6a4c21d35f\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.324257 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.324293 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.325104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.324399 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv\") pod \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\" (UID: \"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c\") " Oct 30 15:15:21.328343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.328302 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.333065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.330623 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.330661 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.333333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.330716 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.330844 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.331124 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.331605 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.332784 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.332877 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.332939 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.333533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.332971 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.340938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.340911 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.341030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.340940 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.341588 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.341562 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.341668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.341612 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:21.341668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.341628 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv" pod "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" (UID: "cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c"). InnerVolumeSpecName "kube-api-access-fjtfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.341668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.341638 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:21.344814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.344786 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4" pod "69c54aae-4a98-45eb-be48-ff6a4c21d35f" (UID: "69c54aae-4a98-45eb-be48-ff6a4c21d35f"). InnerVolumeSpecName "kube-api-access-vsbx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.417578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.417526 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:21.420488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.420464 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:21.424719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.424663 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:21.430255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430232 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430264 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430286 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430320 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430334 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430360 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430373 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430386 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430398 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430409 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430422 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbx4\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-kube-api-access-vsbx4\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430436 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69c54aae-4a98-45eb-be48-ff6a4c21d35f-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430450 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430463 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430478 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtfv\" (UniqueName: \"kubernetes.io/projected/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-kube-api-access-fjtfv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430490 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430504 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.430894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.430521 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69c54aae-4a98-45eb-be48-ff6a4c21d35f-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.532309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.532269 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch\" (UniqueName: \"kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch\") pod \"1d0f5c6b-6606-427b-b356-2275618e69ea\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " Oct 30 15:15:21.532595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.532380 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca\") pod \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " Oct 30 15:15:21.533006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.532974 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca\") pod \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " Oct 30 15:15:21.533858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.533788 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca\") pod \"1d0f5c6b-6606-427b-b356-2275618e69ea\" (UID: \"1d0f5c6b-6606-427b-b356-2275618e69ea\") " Oct 30 15:15:21.534134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.533977 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5\" (UniqueName: \"kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5\") pod \"d799f44c-675e-440f-9c81-8ee0583ca6cd\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " Oct 30 15:15:21.534301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.534281 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4\" (UniqueName: \"kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4\") pod \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\" (UID: \"e8322d9f-3fe5-478b-a966-aa713bdc06e2\") " Oct 30 15:15:21.534508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.534487 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca\") pod \"d799f44c-675e-440f-9c81-8ee0583ca6cd\" (UID: \"d799f44c-675e-440f-9c81-8ee0583ca6cd\") " Oct 30 15:15:21.535317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.535286 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca" pod "e8322d9f-3fe5-478b-a966-aa713bdc06e2" (UID: "e8322d9f-3fe5-478b-a966-aa713bdc06e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.536319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.536291 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca" pod "1d0f5c6b-6606-427b-b356-2275618e69ea" (UID: "1d0f5c6b-6606-427b-b356-2275618e69ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.536935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.536907 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.537019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.536943 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d0f5c6b-6606-427b-b356-2275618e69ea-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.547951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.547923 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca" pod "d799f44c-675e-440f-9c81-8ee0583ca6cd" (UID: "d799f44c-675e-440f-9c81-8ee0583ca6cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.559676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.559630 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4" pod "e8322d9f-3fe5-478b-a966-aa713bdc06e2" (UID: "e8322d9f-3fe5-478b-a966-aa713bdc06e2"). InnerVolumeSpecName "kube-api-access-gqnc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.595742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.595654 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca" pod "e8322d9f-3fe5-478b-a966-aa713bdc06e2" (UID: "e8322d9f-3fe5-478b-a966-aa713bdc06e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:15:21.596529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.596484 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch" pod "1d0f5c6b-6606-427b-b356-2275618e69ea" (UID: "1d0f5c6b-6606-427b-b356-2275618e69ea"). InnerVolumeSpecName "kube-api-access-rbkch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.601469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.601440 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5" pod "d799f44c-675e-440f-9c81-8ee0583ca6cd" (UID: "d799f44c-675e-440f-9c81-8ee0583ca6cd"). InnerVolumeSpecName "kube-api-access-wqgm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.635264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.635236 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:21.637308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.637281 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:21.638076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.638058 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgm5\" (UniqueName: \"kubernetes.io/projected/d799f44c-675e-440f-9c81-8ee0583ca6cd-kube-api-access-wqgm5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.638125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.638079 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqnc4\" (UniqueName: \"kubernetes.io/projected/e8322d9f-3fe5-478b-a966-aa713bdc06e2-kube-api-access-gqnc4\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.638125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.638089 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d799f44c-675e-440f-9c81-8ee0583ca6cd-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.638125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.638098 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbkch\" (UniqueName: \"kubernetes.io/projected/1d0f5c6b-6606-427b-b356-2275618e69ea-kube-api-access-rbkch\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.638125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.638107 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8322d9f-3fe5-478b-a966-aa713bdc06e2-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.740494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.740463 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca\") pod \"b8c26b7b-085b-424e-abe3-5111cb128b1a\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " Oct 30 15:15:21.740820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.740802 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq\" (UniqueName: \"kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq\") pod \"a7edcdb2-552f-4a1c-b36f-dfee06683616\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " Oct 30 15:15:21.741106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.741089 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca\") pod \"a7edcdb2-552f-4a1c-b36f-dfee06683616\" (UID: \"a7edcdb2-552f-4a1c-b36f-dfee06683616\") " Oct 30 15:15:21.741236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.741223 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk\" (UniqueName: \"kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk\") pod \"b8c26b7b-085b-424e-abe3-5111cb128b1a\" (UID: \"b8c26b7b-085b-424e-abe3-5111cb128b1a\") " Oct 30 15:15:21.744200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.743982 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca" pod "a7edcdb2-552f-4a1c-b36f-dfee06683616" (UID: "a7edcdb2-552f-4a1c-b36f-dfee06683616"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.746426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.744504 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca" pod "b8c26b7b-085b-424e-abe3-5111cb128b1a" (UID: "b8c26b7b-085b-424e-abe3-5111cb128b1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:21.764268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.753496 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq" pod "a7edcdb2-552f-4a1c-b36f-dfee06683616" (UID: "a7edcdb2-552f-4a1c-b36f-dfee06683616"). InnerVolumeSpecName "kube-api-access-jnmjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.855238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.854428 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk" pod "b8c26b7b-085b-424e-abe3-5111cb128b1a" (UID: "b8c26b7b-085b-424e-abe3-5111cb128b1a"). InnerVolumeSpecName "kube-api-access-4jbvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:21.866443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.860990 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmjq\" (UniqueName: \"kubernetes.io/projected/a7edcdb2-552f-4a1c-b36f-dfee06683616-kube-api-access-jnmjq\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.866443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.861023 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7edcdb2-552f-4a1c-b36f-dfee06683616-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.866443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.861041 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbvk\" (UniqueName: \"kubernetes.io/projected/b8c26b7b-085b-424e-abe3-5111cb128b1a-kube-api-access-4jbvk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.866443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.861057 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26b7b-085b-424e-abe3-5111cb128b1a-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:21.900924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.900891 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" Oct 30 15:15:21.901089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.900896 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw" event={"ID":"b8c26b7b-085b-424e-abe3-5111cb128b1a","Type":"ContainerDied","Data":"30758e6d1e51b8f168e59c838abc1e6ab3fc5c04fe8644114e7b01851723c47c"} Oct 30 15:15:21.901220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.901206 6554 scope.go:122] "RemoveContainer" containerID="e90506d84ac4238cdb451355b1c293418b795744b35100ccad8c93ca4a2c9fc2" Oct 30 15:15:21.906083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.906069 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" Oct 30 15:15:21.908562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.906197 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss" event={"ID":"d799f44c-675e-440f-9c81-8ee0583ca6cd","Type":"ContainerDied","Data":"f8e11c9d58e9eb861adecc6376dde46c00572df712fdaa80758550001c02d798"} Oct 30 15:15:21.922980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.922945 6554 scope.go:122] "RemoveContainer" containerID="7c0ca001ae8b2cbcfa85c731c6fc9e36712047d4e091da4edef3f30c7283cdfd" Oct 30 15:15:21.924762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.924645 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" event={"ID":"a7edcdb2-552f-4a1c-b36f-dfee06683616","Type":"ContainerDied","Data":"72612c99e8d6901fb47b63fa15dd136021326aaf3da601991ffb4aa8683d32ef"} Oct 30 15:15:21.924762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.924747 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj" Oct 30 15:15:21.926603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.926582 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" Oct 30 15:15:21.926673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.926630 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255" event={"ID":"cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c","Type":"ContainerDied","Data":"a5113a95d9e4414756d81ee98e4aabda4c948a161f1693a402e313836b786deb"} Oct 30 15:15:21.928749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.928728 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" event={"ID":"f6b96720-1681-466e-b94d-07bc26a00523","Type":"ContainerDied","Data":"c23651c28de41b67c7e6fd8960d30aa3a34527a974cebba393f41f52da21efe9"} Oct 30 15:15:21.928890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.928767 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr" Oct 30 15:15:21.935488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.935467 6554 scope.go:122] "RemoveContainer" containerID="03878a41d235f6b9cb657f0b9541f260170a914e4a400b282773d81fd226f8dc" Oct 30 15:15:21.937241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.937111 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" Oct 30 15:15:21.937241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.937146 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c" event={"ID":"e8322d9f-3fe5-478b-a966-aa713bdc06e2","Type":"ContainerDied","Data":"545f3727f188017db37d81472380a624aa17c51b927f806276e827c04299d450"} Oct 30 15:15:21.937241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.937184 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="545f3727f188017db37d81472380a624aa17c51b927f806276e827c04299d450" Oct 30 15:15:21.938643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.938622 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" Oct 30 15:15:21.938815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.938762 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c" event={"ID":"1d0f5c6b-6606-427b-b356-2275618e69ea","Type":"ContainerDied","Data":"eb2e320b32b122a26af89e477d31162630f86f4f017e57af2b3b4335c6490bef"} Oct 30 15:15:21.942335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.941523 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" event={"ID":"69c54aae-4a98-45eb-be48-ff6a4c21d35f","Type":"ContainerDied","Data":"390224bf825b321fac1c8735529cbd1896056ba232d77502fbd1d1fad0253bd3"} Oct 30 15:15:21.942335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.941548 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7" Oct 30 15:15:21.943025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.942983 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" event={"ID":"351672e0-ce2f-4ea0-8987-0b2471da2f6b","Type":"ContainerDied","Data":"ef15e8faa472684e616a26c547d1fb9c46873b77f84ecf032c02ae97039778eb"} Oct 30 15:15:21.943025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.942992 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f" Oct 30 15:15:21.950589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.950573 6554 scope.go:122] "RemoveContainer" containerID="8087cf77a582ba242eee9c06a2f3ce5b8fa18c08f3bfdbeeb17778124e59e506" Oct 30 15:15:21.990046 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.990012 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw"] Oct 30 15:15:21.995270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.995175 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-09e8eb-29889555-zw6pw"] Oct 30 15:15:21.996516 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:21.996497 6554 scope.go:122] "RemoveContainer" containerID="e526ad9992c1e05a1d7fd4f5ef53735f7e0c983885737e00588bc08cd10f531f" Oct 30 15:15:22.009368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.009327 6554 scope.go:122] "RemoveContainer" containerID="75755f810bddb941d1863ad47ed0bec18876c09d465103cc7ea791bc5dd04d4a" Oct 30 15:15:22.022526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.022509 6554 scope.go:122] "RemoveContainer" containerID="437df0c593e859ac80622adc67162523f4f78246f99bc0f9b5ab76f82c7ecf9c" Oct 30 15:15:22.033064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.033043 6554 scope.go:122] "RemoveContainer" containerID="f23b3466273005cd871b62f7d830ac8e14c4ba3a97f1a8372b8f0aba05d8866a" Oct 30 15:15:22.071156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.071124 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp"] Oct 30 15:15:22.115450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.113755 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr"] Oct 30 15:15:22.120195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.120162 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-6b2vr"] Oct 30 15:15:22.158125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.158096 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e987c8-a676-4474-94ad-89b06c778051" path="/var/lib/kubelet/pods/45e987c8-a676-4474-94ad-89b06c778051/volumes" Oct 30 15:15:22.158562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.158547 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7751512c-6473-467a-8fad-4fb8bf588636" path="/var/lib/kubelet/pods/7751512c-6473-467a-8fad-4fb8bf588636/volumes" Oct 30 15:15:22.158946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.158934 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" path="/var/lib/kubelet/pods/98e02f5f-6d5a-4029-8323-f100e719c462/volumes" Oct 30 15:15:22.159594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.159569 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" path="/var/lib/kubelet/pods/b8c26b7b-085b-424e-abe3-5111cb128b1a/volumes" Oct 30 15:15:22.160252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.160231 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b96720-1681-466e-b94d-07bc26a00523" path="/var/lib/kubelet/pods/f6b96720-1681-466e-b94d-07bc26a00523/volumes" Oct 30 15:15:22.160715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.160697 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss"] Oct 30 15:15:22.166896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.166870 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-97ead4-29889555-gz7ss"] Oct 30 15:15:22.212804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.212773 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj"] Oct 30 15:15:22.222897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.222869 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-147bdb-29889555-647jj"] Oct 30 15:15:22.256036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.256006 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255"] Oct 30 15:15:22.260084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.260060 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-nw255"] Oct 30 15:15:22.262794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.262774 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:22.298142 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.298119 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c"] Oct 30 15:15:22.305925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.305900 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e64e05-29889555-cpd2c"] Oct 30 15:15:22.371140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.371070 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca\") pod \"640e515c-16b2-49d4-beea-43ee099f7f09\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " Oct 30 15:15:22.371316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.371303 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td\" (UniqueName: \"kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td\") pod \"640e515c-16b2-49d4-beea-43ee099f7f09\" (UID: \"640e515c-16b2-49d4-beea-43ee099f7f09\") " Oct 30 15:15:22.371497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.371472 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca" pod "640e515c-16b2-49d4-beea-43ee099f7f09" (UID: "640e515c-16b2-49d4-beea-43ee099f7f09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:22.371945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.371926 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640e515c-16b2-49d4-beea-43ee099f7f09-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:22.373496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.373473 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td" pod "640e515c-16b2-49d4-beea-43ee099f7f09" (UID: "640e515c-16b2-49d4-beea-43ee099f7f09"). InnerVolumeSpecName "kube-api-access-gj6td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:22.382770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.382749 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f"] Oct 30 15:15:22.391057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.391028 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-5sm8f"] Oct 30 15:15:22.440997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.440970 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7"] Oct 30 15:15:22.450613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.450591 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7acff45-openshift-default-6f9b7c6cb9-2wrx7"] Oct 30 15:15:22.473864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.473840 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj6td\" (UniqueName: \"kubernetes.io/projected/640e515c-16b2-49d4-beea-43ee099f7f09-kube-api-access-gj6td\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:22.958520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.958490 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" Oct 30 15:15:22.958520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.958497 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp" event={"ID":"640e515c-16b2-49d4-beea-43ee099f7f09","Type":"ContainerDied","Data":"d4be26719086f206a4052cf5e8953e48fb1fee38ee06470828d603d6c489b6f3"} Oct 30 15:15:22.959031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:22.958555 6554 scope.go:122] "RemoveContainer" containerID="cae1d32fd3069c8e1df945b3fcfe3b435956d7217703224b9d83c52169541a98" Oct 30 15:15:23.017604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:23.017576 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp"] Oct 30 15:15:23.021194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:23.021167 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-a65941-29889555-kd6kp"] Oct 30 15:15:24.161457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.161422 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" path="/var/lib/kubelet/pods/1d0f5c6b-6606-427b-b356-2275618e69ea/volumes" Oct 30 15:15:24.162211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.162188 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" path="/var/lib/kubelet/pods/351672e0-ce2f-4ea0-8987-0b2471da2f6b/volumes" Oct 30 15:15:24.162822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.162800 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" path="/var/lib/kubelet/pods/640e515c-16b2-49d4-beea-43ee099f7f09/volumes" Oct 30 15:15:24.163988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.163965 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" path="/var/lib/kubelet/pods/69c54aae-4a98-45eb-be48-ff6a4c21d35f/volumes" Oct 30 15:15:24.164611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.164577 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" path="/var/lib/kubelet/pods/a7edcdb2-552f-4a1c-b36f-dfee06683616/volumes" Oct 30 15:15:24.165310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.165288 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" path="/var/lib/kubelet/pods/cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c/volumes" Oct 30 15:15:24.166294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:24.166273 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" path="/var/lib/kubelet/pods/d799f44c-675e-440f-9c81-8ee0583ca6cd/volumes" Oct 30 15:15:25.228466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.228433 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:15:25.228850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.228480 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:15:25.992125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.992069 6554 generic.go:356] "Generic (PLEG): container finished" podID="ab402479-669a-4335-8c47-b7bce678d1d8" containerID="d8ab73cdd61ed7866151621d541b96bc3cee4ceec1d201b46b9d4aa1fda759fb" exitCode=0 Oct 30 15:15:25.992125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.992113 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" event={"ID":"ab402479-669a-4335-8c47-b7bce678d1d8","Type":"ContainerDied","Data":"d8ab73cdd61ed7866151621d541b96bc3cee4ceec1d201b46b9d4aa1fda759fb"} Oct 30 15:15:25.992528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.992510 6554 scope.go:122] "RemoveContainer" containerID="d8ab73cdd61ed7866151621d541b96bc3cee4ceec1d201b46b9d4aa1fda759fb" Oct 30 15:15:25.994225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.994196 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:25.994290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.994267 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:25.994326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:25.994297 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:26.918665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:26.918583 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s"] Oct 30 15:15:26.919145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:26.918799 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" containerID="cri-o://1ad0475590c1dfc000f4ee3ae7a282b9e46e0cad76a4873d190f9e6e6e8c1a8e" gracePeriod=30 Oct 30 15:15:27.002009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:27.001978 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" event={"ID":"ab402479-669a-4335-8c47-b7bce678d1d8","Type":"ContainerStarted","Data":"0dc4158959a391525d0aa43805a528cc28f96b54a03cca2aa01e8423f2b83866"} Oct 30 15:15:27.195096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:27.195000 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt"] Oct 30 15:15:27.237303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:27.237268 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7"] Oct 30 15:15:27.407781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:27.407752 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:15:27.407969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:27.407944 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:15:27.408023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:27.407989 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:15:28.180933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:28.180813 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:15:28.180933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:28.180883 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:15:28.408510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:28.408475 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:15:28.408680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:28.408540 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:15:29.408286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:29.408253 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:15:29.408713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:29.408300 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:15:30.407878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:30.407835 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:15:30.408115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:30.407900 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:15:30.684428 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:30.677540 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:31.408056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:31.408025 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:15:31.408234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:31.408102 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:15:32.102338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.102304 6554 generic.go:356] "Generic (PLEG): container finished" podID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerID="1ad0475590c1dfc000f4ee3ae7a282b9e46e0cad76a4873d190f9e6e6e8c1a8e" exitCode=0 Oct 30 15:15:32.102845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.102393 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" event={"ID":"1aab4ca9-aa15-4ea5-9919-6585d9bf6718","Type":"ContainerDied","Data":"1ad0475590c1dfc000f4ee3ae7a282b9e46e0cad76a4873d190f9e6e6e8c1a8e"} Oct 30 15:15:32.102845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.102457 6554 scope.go:122] "RemoveContainer" containerID="4447ccfff210a25a263753a91d4f737cd118225ba604a880ed746bf9b5c6c19d" Oct 30 15:15:32.195877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.195763 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" podUID="467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" containerName="payload-processing" containerID="cri-o://134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3" gracePeriod=25 Oct 30 15:15:32.237762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.237730 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" podUID="e7a1c21a-b72f-43c9-8142-142465fe33cc" containerName="payload-pre-processing" containerID="cri-o://eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77" gracePeriod=25 Oct 30 15:15:32.390248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.390222 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 15:15:32.409403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.408932 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 15:15:32.409403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.409001 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 15:15:32.494393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494027 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494086 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494260 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494338 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494421 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494488 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494533 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494584 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.494664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.494648 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket\") pod \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\" (UID: \"1aab4ca9-aa15-4ea5-9919-6585d9bf6718\") " Oct 30 15:15:32.495970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.495944 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:32.496859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.496830 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:32.497445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.497160 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:32.497978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.497948 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:32.498074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.498016 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:32.500301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.500275 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "kube-api-access-284zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:32.500994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.500967 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:32.501454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.501432 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:32.502176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.502144 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo" pod "1aab4ca9-aa15-4ea5-9919-6585d9bf6718" (UID: "1aab4ca9-aa15-4ea5-9919-6585d9bf6718"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:32.584319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.583999 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597395 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597427 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597440 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-284zb\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-kube-api-access-284zb\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597454 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597466 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597478 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597489 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597502 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.598049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.597517 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aab4ca9-aa15-4ea5-9919-6585d9bf6718-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.603047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.603016 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:15:32.698704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.698667 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume\") pod \"e7a1c21a-b72f-43c9-8142-142465fe33cc\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " Oct 30 15:15:32.698905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.698811 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume\") pod \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " Oct 30 15:15:32.698905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.698885 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6\" (UniqueName: \"kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6\") pod \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\" (UID: \"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b\") " Oct 30 15:15:32.699019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.699002 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9\" (UniqueName: \"kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9\") pod \"e7a1c21a-b72f-43c9-8142-142465fe33cc\" (UID: \"e7a1c21a-b72f-43c9-8142-142465fe33cc\") " Oct 30 15:15:32.699695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.699661 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume" pod "467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" (UID: "467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:32.699877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.699858 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.700072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.700050 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume" pod "e7a1c21a-b72f-43c9-8142-142465fe33cc" (UID: "e7a1c21a-b72f-43c9-8142-142465fe33cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:32.702257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.702198 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9" pod "e7a1c21a-b72f-43c9-8142-142465fe33cc" (UID: "e7a1c21a-b72f-43c9-8142-142465fe33cc"). InnerVolumeSpecName "kube-api-access-sjnd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:32.702997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.702974 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6" pod "467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" (UID: "467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b"). InnerVolumeSpecName "kube-api-access-wlgd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:32.801769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.801691 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjnd9\" (UniqueName: \"kubernetes.io/projected/e7a1c21a-b72f-43c9-8142-142465fe33cc-kube-api-access-sjnd9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.801769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.801718 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7a1c21a-b72f-43c9-8142-142465fe33cc-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:32.801769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:32.801727 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlgd6\" (UniqueName: \"kubernetes.io/projected/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b-kube-api-access-wlgd6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:33.131753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.131667 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" Oct 30 15:15:33.131753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.131689 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s" event={"ID":"1aab4ca9-aa15-4ea5-9919-6585d9bf6718","Type":"ContainerDied","Data":"9faafdf238a0fa9a4ae1fbcc318798348f19ed3c2b2556d006368c6348542a2c"} Oct 30 15:15:33.132286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.131804 6554 scope.go:122] "RemoveContainer" containerID="1ad0475590c1dfc000f4ee3ae7a282b9e46e0cad76a4873d190f9e6e6e8c1a8e" Oct 30 15:15:33.133545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.133516 6554 generic.go:356] "Generic (PLEG): container finished" podID="e7a1c21a-b72f-43c9-8142-142465fe33cc" containerID="eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77" exitCode=0 Oct 30 15:15:33.133667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.133596 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" Oct 30 15:15:33.133667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.133600 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" event={"ID":"e7a1c21a-b72f-43c9-8142-142465fe33cc","Type":"ContainerDied","Data":"eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77"} Oct 30 15:15:33.133667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.133655 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7" event={"ID":"e7a1c21a-b72f-43c9-8142-142465fe33cc","Type":"ContainerDied","Data":"285dfdd19bf0aacd89f61c08ec5287fc286e929a9385c62d6cbe5bd927a23c3a"} Oct 30 15:15:33.139745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.139714 6554 generic.go:356] "Generic (PLEG): container finished" podID="467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" containerID="134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3" exitCode=0 Oct 30 15:15:33.139917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.139764 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" event={"ID":"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b","Type":"ContainerDied","Data":"134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3"} Oct 30 15:15:33.139917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.139777 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" Oct 30 15:15:33.139917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.139802 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt" event={"ID":"467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b","Type":"ContainerDied","Data":"9c6895498bb0b9f411cc2c2d7a557d94ca4e3023b99d4406582a69081235b1c1"} Oct 30 15:15:33.151925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.149569 6554 scope.go:122] "RemoveContainer" containerID="eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77" Oct 30 15:15:33.162880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.162861 6554 scope.go:122] "RemoveContainer" containerID="eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77" Oct 30 15:15:33.163173 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:33.163152 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77\": container with ID starting with eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77 not found: ID does not exist" containerID="eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77" Oct 30 15:15:33.163246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.163187 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77"} err="failed to get container status \"eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77\": rpc error: code = NotFound desc = could not find container \"eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77\": container with ID starting with eaf097e42cf63151c53f223dd07c528b6c1602dc8e519c4162e84b4eebecbd77 not found: ID does not exist" Oct 30 15:15:33.163246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.163212 6554 scope.go:122] "RemoveContainer" containerID="134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3" Oct 30 15:15:33.172751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.172731 6554 scope.go:122] "RemoveContainer" containerID="134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3" Oct 30 15:15:33.173021 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:33.172998 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3\": container with ID starting with 134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3 not found: ID does not exist" containerID="134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3" Oct 30 15:15:33.173117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.173025 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3"} err="failed to get container status \"134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3\": rpc error: code = NotFound desc = could not find container \"134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3\": container with ID starting with 134dd89642c8070b62714e00dea707ea12b253e1285dc0a6d893a51f11f90af3 not found: ID does not exist" Oct 30 15:15:33.222748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.222716 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7"] Oct 30 15:15:33.228408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.228385 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-1b7814-5689c7f4c9-65fx7"] Oct 30 15:15:33.264267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.264243 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt"] Oct 30 15:15:33.269883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.269859 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1b7814-78c67bff76-v6mpt"] Oct 30 15:15:33.307070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.307042 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s"] Oct 30 15:15:33.312008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.311987 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-8tv5s"] Oct 30 15:15:33.412935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.412913 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:15:33.413211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.413188 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:15:33.413931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:33.413913 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:15:34.159057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:34.159025 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" path="/var/lib/kubelet/pods/1aab4ca9-aa15-4ea5-9919-6585d9bf6718/volumes" Oct 30 15:15:34.159590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:34.159568 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" path="/var/lib/kubelet/pods/467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b/volumes" Oct 30 15:15:34.160139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:34.160118 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7a1c21a-b72f-43c9-8142-142465fe33cc" path="/var/lib/kubelet/pods/e7a1c21a-b72f-43c9-8142-142465fe33cc/volumes" Oct 30 15:15:35.311632 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:35.311591 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:36.263863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.263829 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s"] Oct 30 15:15:36.264628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264612 6554 cpu_manager.go:414] "containerMap: removing container" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264630 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264643 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264648 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264656 6554 cpu_manager.go:414] "containerMap: removing container" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264661 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264669 6554 cpu_manager.go:414] "containerMap: removing container" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264673 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264679 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.264681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264683 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264694 6554 cpu_manager.go:414] "containerMap: removing container" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264698 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264704 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264708 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264716 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264720 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264727 6554 cpu_manager.go:414] "containerMap: removing container" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264732 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264745 6554 cpu_manager.go:414] "containerMap: removing container" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264750 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264756 6554 cpu_manager.go:414] "containerMap: removing container" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264761 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264767 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e8322d9f-3fe5-478b-a966-aa713bdc06e2" containerName="revoke-keys" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264771 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8322d9f-3fe5-478b-a966-aa713bdc06e2" containerName="revoke-keys" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264779 6554 cpu_manager.go:414] "containerMap: removing container" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264784 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264791 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264796 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264802 6554 cpu_manager.go:414] "containerMap: removing container" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264807 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264813 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264817 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264825 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264829 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264836 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264842 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264857 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264862 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264871 6554 cpu_manager.go:414] "containerMap: removing container" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264876 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264883 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264887 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264895 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264899 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264907 6554 cpu_manager.go:414] "containerMap: removing container" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264913 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264921 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264926 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264936 6554 cpu_manager.go:414] "containerMap: removing container" podUID="467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" containerName="payload-processing" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264941 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" containerName="payload-processing" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264950 6554 cpu_manager.go:414] "containerMap: removing container" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264956 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264966 6554 cpu_manager.go:414] "containerMap: removing container" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" Oct 30 15:15:36.264954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264970 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264983 6554 cpu_manager.go:414] "containerMap: removing container" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.264989 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265003 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e7a1c21a-b72f-43c9-8142-142465fe33cc" containerName="payload-pre-processing" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265010 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7a1c21a-b72f-43c9-8142-142465fe33cc" containerName="payload-pre-processing" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265025 6554 cpu_manager.go:414] "containerMap: removing container" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265030 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265159 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265166 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265173 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265180 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265185 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265192 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265198 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265204 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265210 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265217 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265223 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265228 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265232 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b96720-1681-466e-b94d-07bc26a00523" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265239 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265244 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="467dcb64-f081-4c2e-8cf8-ac6a2dfdbe9b" containerName="payload-processing" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265251 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c54aae-4a98-45eb-be48-ff6a4c21d35f" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265257 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6d4be6-01e2-4bcb-ad1e-8c5fba1f6f6c" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265263 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265269 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7a1c21a-b72f-43c9-8142-142465fe33cc" containerName="payload-pre-processing" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265276 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265282 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265289 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="351672e0-ce2f-4ea0-8987-0b2471da2f6b" containerName="istio-proxy" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265293 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265300 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265305 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265310 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265318 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="640e515c-16b2-49d4-beea-43ee099f7f09" containerName="cleanup" Oct 30 15:15:36.266242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.265325 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8322d9f-3fe5-478b-a966-aa713bdc06e2" containerName="revoke-keys" Oct 30 15:15:36.270472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.270452 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.290003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.289979 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s"] Oct 30 15:15:36.399795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.399761 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9254b0-openshift-default-dockercfg-r8vd9\"" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403287 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403411 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403496 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403610 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403635 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403675 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403699 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw98t\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403738 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.404091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.403793 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.504981 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505109 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505135 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505173 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505195 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw98t\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505226 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505276 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505332 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505444 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505592 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505652 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.505873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.505846 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.506055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.506027 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.506253 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.506230 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.508028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.507694 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.508028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.508000 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.530009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.529950 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.530009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.529998 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw98t\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t\") pod \"e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.587302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.587281 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:36.912687 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:36.912650 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cf74c6_cd6a_46a3_b89b_1582d122ceae.slice/crio-c41dee388c194069d59c33083db4fc05664793d53943b0d8703d29df712fa582 WatchSource:0}: Error finding container c41dee388c194069d59c33083db4fc05664793d53943b0d8703d29df712fa582: Status 404 returned error can't find the container with id c41dee388c194069d59c33083db4fc05664793d53943b0d8703d29df712fa582 Oct 30 15:15:36.913066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.913042 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s"] Oct 30 15:15:36.915538 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.915505 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:36.915624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.915567 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:36.915624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:36.915614 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:37.182950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:37.182853 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" event={"ID":"c5cf74c6-cd6a-46a3-b89b-1582d122ceae","Type":"ContainerStarted","Data":"b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd"} Oct 30 15:15:37.182950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:37.182923 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" event={"ID":"c5cf74c6-cd6a-46a3-b89b-1582d122ceae","Type":"ContainerStarted","Data":"c41dee388c194069d59c33083db4fc05664793d53943b0d8703d29df712fa582"} Oct 30 15:15:37.231597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:37.231543 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podStartSLOduration=1.231527918 podStartE2EDuration="1.231527918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:37.231336824 +0000 UTC m=+1641.190378230" watchObservedRunningTime="2026-10-30 15:15:37.231527918 +0000 UTC m=+1641.190569324" Oct 30 15:15:37.587461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:37.587382 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:37.588709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:37.588687 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:15:37.588773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:37.588744 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 15:15:38.588249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:38.588212 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:15:38.588668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:38.588267 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 15:15:38.790286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:38.790246 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9"] Oct 30 15:15:38.790762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:38.790725 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" podUID="99094cec-d5d2-4e24-a57c-368572c13477" containerName="maas-api" containerID="cri-o://b61999cd6454ec6fe8a8cc4ad74e653f29b10889c3280a89c701d1f4c8142da8" gracePeriod=30 Oct 30 15:15:39.329641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.328991 6554 generic.go:356] "Generic (PLEG): container finished" podID="99094cec-d5d2-4e24-a57c-368572c13477" containerID="b61999cd6454ec6fe8a8cc4ad74e653f29b10889c3280a89c701d1f4c8142da8" exitCode=0 Oct 30 15:15:39.330176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.330147 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" event={"ID":"99094cec-d5d2-4e24-a57c-368572c13477","Type":"ContainerDied","Data":"b61999cd6454ec6fe8a8cc4ad74e653f29b10889c3280a89c701d1f4c8142da8"} Oct 30 15:15:39.406636 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:39.406543 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:15:39.413617 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.413590 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:15:39.508275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.508162 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca\") pod \"99094cec-d5d2-4e24-a57c-368572c13477\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " Oct 30 15:15:39.508275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.508254 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs\") pod \"99094cec-d5d2-4e24-a57c-368572c13477\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " Oct 30 15:15:39.508545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.508394 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\") pod \"99094cec-d5d2-4e24-a57c-368572c13477\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " Oct 30 15:15:39.508545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.508462 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6\" (UniqueName: \"kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6\") pod \"99094cec-d5d2-4e24-a57c-368572c13477\" (UID: \"99094cec-d5d2-4e24-a57c-368572c13477\") " Oct 30 15:15:39.513368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.513292 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca" pod "99094cec-d5d2-4e24-a57c-368572c13477" (UID: "99094cec-d5d2-4e24-a57c-368572c13477"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:39.541462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.541419 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs" pod "99094cec-d5d2-4e24-a57c-368572c13477" (UID: "99094cec-d5d2-4e24-a57c-368572c13477"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:15:39.541462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.541447 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls" pod "99094cec-d5d2-4e24-a57c-368572c13477" (UID: "99094cec-d5d2-4e24-a57c-368572c13477"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:15:39.541865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.541839 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6" pod "99094cec-d5d2-4e24-a57c-368572c13477" (UID: "99094cec-d5d2-4e24-a57c-368572c13477"). InnerVolumeSpecName "kube-api-access-jxzb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:39.606770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.602482 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:15:39.606770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.602538 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 15:15:39.617383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.616977 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:39.617383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.617007 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxzb6\" (UniqueName: \"kubernetes.io/projected/99094cec-d5d2-4e24-a57c-368572c13477-kube-api-access-jxzb6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:39.617383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.617017 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99094cec-d5d2-4e24-a57c-368572c13477-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:39.617383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:39.617026 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99094cec-d5d2-4e24-a57c-368572c13477-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:40.083547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.083514 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb"] Oct 30 15:15:40.084393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084372 6554 cpu_manager.go:414] "containerMap: removing container" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:40.084393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084392 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084410 6554 cpu_manager.go:414] "containerMap: removing container" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084415 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084427 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084432 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aab4ca9-aa15-4ea5-9919-6585d9bf6718" containerName="istio-proxy" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084451 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084455 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084465 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084469 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084498 6554 cpu_manager.go:414] "containerMap: removing container" podUID="99094cec-d5d2-4e24-a57c-368572c13477" containerName="maas-api" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084502 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99094cec-d5d2-4e24-a57c-368572c13477" containerName="maas-api" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084511 6554 cpu_manager.go:414] "containerMap: removing container" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:40.084557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084515 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084613 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="99094cec-d5d2-4e24-a57c-368572c13477" containerName="maas-api" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084622 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e987c8-a676-4474-94ad-89b06c778051" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084631 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7edcdb2-552f-4a1c-b36f-dfee06683616" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084638 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e02f5f-6d5a-4029-8323-f100e719c462" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084645 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="d799f44c-675e-440f-9c81-8ee0583ca6cd" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084651 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7751512c-6473-467a-8fad-4fb8bf588636" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084659 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c26b7b-085b-424e-abe3-5111cb128b1a" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084667 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="048b6a4e-fbee-489e-a415-698df68d4bb5" containerName="cleanup" Oct 30 15:15:40.084941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.084674 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0f5c6b-6606-427b-b356-2275618e69ea" containerName="cleanup" Oct 30 15:15:40.088558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.088540 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.105099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.105075 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-34218d\"" Oct 30 15:15:40.105682 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.105276 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-34218d-dockercfg-222dk\"" Oct 30 15:15:40.107689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.107669 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb"] Oct 30 15:15:40.128324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.128299 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.128469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.128379 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7txg\" (UniqueName: \"kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg\") pod \"payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.271953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.264437 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.271953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.264575 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7txg\" (UniqueName: \"kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg\") pod \"payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.327092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.327053 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.387433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.374210 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5"] Oct 30 15:15:40.412414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.400498 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.423384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.413137 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5"] Oct 30 15:15:40.448071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.448041 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7txg\" (UniqueName: \"kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg\") pod \"payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.491959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.491927 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" Oct 30 15:15:40.492186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.492157 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9" event={"ID":"99094cec-d5d2-4e24-a57c-368572c13477","Type":"ContainerDied","Data":"3cc3351501afa9338b4d3a125adbcdd2f933d14254b0a9da78ba1ed72b33cbb1"} Oct 30 15:15:40.492245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.492220 6554 scope.go:122] "RemoveContainer" containerID="b61999cd6454ec6fe8a8cc4ad74e653f29b10889c3280a89c701d1f4c8142da8" Oct 30 15:15:40.542934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.542906 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.543143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.543124 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsc8k\" (UniqueName: \"kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k\") pod \"payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.578371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.562400 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9"] Oct 30 15:15:40.580882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.580853 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-216204-54f54c64c8-qcpr9"] Oct 30 15:15:40.588793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.588766 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:15:40.588871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.588829 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 15:15:40.644666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.644610 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsc8k\" (UniqueName: \"kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k\") pod \"payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.645019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.644810 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.645572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.645543 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.680585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.680561 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsc8k\" (UniqueName: \"kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k\") pod \"payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.702822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.702800 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:40.797866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:40.793300 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:40.825121 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:40.824885 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:41.085007 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.084975 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb"] Oct 30 15:15:41.085452 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:41.085415 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f03875_b078_4c81_b512_26e085ff6388.slice/crio-72d58a2a94d60471ef4820d134c8738435526b93d5994f315e0bc8773dd07efe WatchSource:0}: Error finding container 72d58a2a94d60471ef4820d134c8738435526b93d5994f315e0bc8773dd07efe: Status 404 returned error can't find the container with id 72d58a2a94d60471ef4820d134c8738435526b93d5994f315e0bc8773dd07efe Oct 30 15:15:41.152217 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:41.152181 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba806c4_36f9_4df7_8430_27caa0178955.slice/crio-d7678c82c1f953d822bdffa79c3201b61298627ebbfce1eaf4e47d3074c4db92 WatchSource:0}: Error finding container d7678c82c1f953d822bdffa79c3201b61298627ebbfce1eaf4e47d3074c4db92: Status 404 returned error can't find the container with id d7678c82c1f953d822bdffa79c3201b61298627ebbfce1eaf4e47d3074c4db92 Oct 30 15:15:41.152321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.152270 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5"] Oct 30 15:15:41.502693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.502612 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" event={"ID":"9ba806c4-36f9-4df7-8430-27caa0178955","Type":"ContainerStarted","Data":"a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb"} Oct 30 15:15:41.502693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.502682 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" event={"ID":"9ba806c4-36f9-4df7-8430-27caa0178955","Type":"ContainerStarted","Data":"d7678c82c1f953d822bdffa79c3201b61298627ebbfce1eaf4e47d3074c4db92"} Oct 30 15:15:41.502913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.502810 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:41.504423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.504395 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" event={"ID":"c7f03875-b078-4c81-b512-26e085ff6388","Type":"ContainerStarted","Data":"0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5"} Oct 30 15:15:41.504574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.504441 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" event={"ID":"c7f03875-b078-4c81-b512-26e085ff6388","Type":"ContainerStarted","Data":"72d58a2a94d60471ef4820d134c8738435526b93d5994f315e0bc8773dd07efe"} Oct 30 15:15:41.504574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.504482 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:41.547798 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.547751 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" podStartSLOduration=1.547739323 podStartE2EDuration="1.547739323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:41.545664453 +0000 UTC m=+1645.504705859" watchObservedRunningTime="2026-10-30 15:15:41.547739323 +0000 UTC m=+1645.506780756" Oct 30 15:15:41.588372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.588315 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:15:41.588507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.588384 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 15:15:41.600585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:41.600535 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" podStartSLOduration=1.6005225250000001 podStartE2EDuration="1.600522525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:41.59908885 +0000 UTC m=+1645.558130254" watchObservedRunningTime="2026-10-30 15:15:41.600522525 +0000 UTC m=+1645.559563930" Oct 30 15:15:42.186407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:42.186332 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99094cec-d5d2-4e24-a57c-368572c13477" path="/var/lib/kubelet/pods/99094cec-d5d2-4e24-a57c-368572c13477/volumes" Oct 30 15:15:42.588489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:42.588414 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 15:15:42.588489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:42.588465 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 15:15:43.640451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:43.639967 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:43.641015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:43.640988 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:43.642119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:43.642087 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:15:47.356109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:47.331033 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp"] Oct 30 15:15:47.356109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:47.342216 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" containerID="cri-o://2ae82d7cb359284431eec182be8d013da4d9660302fc98e87ef3f2728c6d9cdf" gracePeriod=30 Oct 30 15:15:48.604220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.604177 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv"] Oct 30 15:15:48.611836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.611807 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.629785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.629755 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv"] Oct 30 15:15:48.638732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.638709 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3bf5d2-openshift-default-dockercfg-z5mdz\"" Oct 30 15:15:48.767449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767402 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767512 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767594 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767621 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767732 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hshbw\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767768 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767906 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.767974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.767961 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.768064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.768053 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.873565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.873455 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.873914 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.873887 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874069 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874146 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874248 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874292 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874329 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hshbw\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874381 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.874655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.874556 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.875293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.875262 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.875938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.875912 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.876300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.876278 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.876633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.876608 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.876886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.876865 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.882871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.881455 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.898696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.898661 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.902467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.902444 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.908407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.906915 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hshbw\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw\") pod \"e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:48.927636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:48.927191 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:49.271031 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:49.271000 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ed6f78_cca3_4e47_b41b_9f0e2200ce88.slice/crio-d08dd0ea3db2d7f0a5a2c30643914a88a6d1c248952b305d90145a1e40271622 WatchSource:0}: Error finding container d08dd0ea3db2d7f0a5a2c30643914a88a6d1c248952b305d90145a1e40271622: Status 404 returned error can't find the container with id d08dd0ea3db2d7f0a5a2c30643914a88a6d1c248952b305d90145a1e40271622 Oct 30 15:15:49.272196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.272173 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv"] Oct 30 15:15:49.274113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.274077 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:49.274206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.274141 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:49.274206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.274171 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:15:49.586806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.586717 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" event={"ID":"46ed6f78-cca3-4e47-b41b-9f0e2200ce88","Type":"ContainerStarted","Data":"28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747"} Oct 30 15:15:49.586806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.586770 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" event={"ID":"46ed6f78-cca3-4e47-b41b-9f0e2200ce88","Type":"ContainerStarted","Data":"d08dd0ea3db2d7f0a5a2c30643914a88a6d1c248952b305d90145a1e40271622"} Oct 30 15:15:49.642469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.642416 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podStartSLOduration=1.642402352 podStartE2EDuration="1.642402352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:49.639257141 +0000 UTC m=+1653.598298547" watchObservedRunningTime="2026-10-30 15:15:49.642402352 +0000 UTC m=+1653.601443756" Oct 30 15:15:49.927848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:49.927810 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:50.386547 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:50.386519 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:50.845384 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:15:50.845277 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:15:50.934055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:50.934028 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:51.602061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:51.602030 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:51.602996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:51.602971 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:15:52.521133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.521092 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:15:52.521704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.521185 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:15:52.623442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.623257 6554 generic.go:356] "Generic (PLEG): container finished" podID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerID="2ae82d7cb359284431eec182be8d013da4d9660302fc98e87ef3f2728c6d9cdf" exitCode=0 Oct 30 15:15:52.624611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.624420 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" event={"ID":"e45ed971-cdbc-4e3f-91ec-6f6f442f0800","Type":"ContainerDied","Data":"2ae82d7cb359284431eec182be8d013da4d9660302fc98e87ef3f2728c6d9cdf"} Oct 30 15:15:52.679211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.679144 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:15:52.832507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832471 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.832701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832538 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.832776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832717 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.832839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832788 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.832904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832830 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.832904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832845 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.833017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832958 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.833017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.832963 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.833126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.833070 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.833185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.833128 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.833185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.833133 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.833185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.833163 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo\") pod \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\" (UID: \"e45ed971-cdbc-4e3f-91ec-6f6f442f0800\") " Oct 30 15:15:52.833367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.833179 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:15:52.833676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.833638 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.834057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.834032 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.834155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.834064 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.834155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.834079 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.834155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.834092 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.834155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.834105 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.835580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.835559 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:52.835812 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.835790 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:15:52.836239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.836196 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "kube-api-access-vpkf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:15:52.836239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.836214 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo" pod "e45ed971-cdbc-4e3f-91ec-6f6f442f0800" (UID: "e45ed971-cdbc-4e3f-91ec-6f6f442f0800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:15:52.936125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.936062 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkf8\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-kube-api-access-vpkf8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.936125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.936092 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.936125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.936102 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:52.936125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:52.936111 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e45ed971-cdbc-4e3f-91ec-6f6f442f0800-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:15:53.790370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:53.789579 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" Oct 30 15:15:53.803813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:53.803769 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp" event={"ID":"e45ed971-cdbc-4e3f-91ec-6f6f442f0800","Type":"ContainerDied","Data":"0c4958d1bedefbaada542e077abfc288d27c384bc7634ec457c6f25adeb52ce3"} Oct 30 15:15:53.804124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:53.804109 6554 scope.go:122] "RemoveContainer" containerID="2ae82d7cb359284431eec182be8d013da4d9660302fc98e87ef3f2728c6d9cdf" Oct 30 15:15:53.888403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:53.888370 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp"] Oct 30 15:15:53.904132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:53.904102 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-216204-openshift-default-7564567d9c-ng7lp"] Oct 30 15:15:54.157037 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.156997 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" path="/var/lib/kubelet/pods/e45ed971-cdbc-4e3f-91ec-6f6f442f0800/volumes" Oct 30 15:15:54.157446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.157430 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l"] Oct 30 15:15:54.158163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.158149 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" Oct 30 15:15:54.158206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.158165 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" Oct 30 15:15:54.158308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.158299 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e45ed971-cdbc-4e3f-91ec-6f6f442f0800" containerName="istio-proxy" Oct 30 15:15:54.163724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.163704 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.180843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.180821 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-3bf5d2\"" Oct 30 15:15:54.181936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.181916 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-3bf5d2-dockercfg-c6zns\"" Oct 30 15:15:54.184586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.184565 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l"] Oct 30 15:15:54.274377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.269537 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.274377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.269849 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lps79\" (UniqueName: \"kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79\") pod \"payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.384380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.371743 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.384380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.373156 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.384380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.378532 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lps79\" (UniqueName: \"kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79\") pod \"payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.384380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.380868 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46"] Oct 30 15:15:54.398371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.388994 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46"] Oct 30 15:15:54.398371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.389197 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.422627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.416361 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lps79\" (UniqueName: \"kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79\") pod \"payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.478590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.478559 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:54.480384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.480343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzgkw\" (UniqueName: \"kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw\") pod \"payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.480527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.480425 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.582693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.582331 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzgkw\" (UniqueName: \"kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw\") pod \"payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.582693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.582425 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.584077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.583696 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.612068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.612042 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzgkw\" (UniqueName: \"kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw\") pod \"payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.742724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.742665 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:54.803210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:54.803153 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l"] Oct 30 15:15:55.060642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.060615 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46"] Oct 30 15:15:55.060747 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:15:55.060669 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bd7df0_55e9_4aea_ad3e_324dd477c83c.slice/crio-959bd85eb0c2659aeab89dfedcc592d4c64a8eec235a6e645f5c7c57fdab3f95 WatchSource:0}: Error finding container 959bd85eb0c2659aeab89dfedcc592d4c64a8eec235a6e645f5c7c57fdab3f95: Status 404 returned error can't find the container with id 959bd85eb0c2659aeab89dfedcc592d4c64a8eec235a6e645f5c7c57fdab3f95 Oct 30 15:15:55.821291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.821256 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" event={"ID":"43bd7df0-55e9-4aea-ad3e-324dd477c83c","Type":"ContainerStarted","Data":"60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289"} Oct 30 15:15:55.821803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.821316 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" event={"ID":"43bd7df0-55e9-4aea-ad3e-324dd477c83c","Type":"ContainerStarted","Data":"959bd85eb0c2659aeab89dfedcc592d4c64a8eec235a6e645f5c7c57fdab3f95"} Oct 30 15:15:55.821803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.821346 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:15:55.823126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.823101 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" event={"ID":"654bc5f1-11ca-40ce-b688-471db30261be","Type":"ContainerStarted","Data":"e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f"} Oct 30 15:15:55.823260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.823141 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" event={"ID":"654bc5f1-11ca-40ce-b688-471db30261be","Type":"ContainerStarted","Data":"00d6627df904da775d6fb8eb7767053de18042dbbf3cf288c2dc5f7dd4bc635a"} Oct 30 15:15:55.823260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.823163 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:15:55.871238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.871191 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" podStartSLOduration=1.871178484 podStartE2EDuration="1.871178484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:55.867220493 +0000 UTC m=+1659.826261900" watchObservedRunningTime="2026-10-30 15:15:55.871178484 +0000 UTC m=+1659.830219889" Oct 30 15:15:55.915570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:15:55.915528 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" podStartSLOduration=1.915511264 podStartE2EDuration="1.915511264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:15:55.910749976 +0000 UTC m=+1659.869791418" watchObservedRunningTime="2026-10-30 15:15:55.915511264 +0000 UTC m=+1659.874552669" Oct 30 15:16:00.955002 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:00.953166 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:16:05.312619 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:05.312579 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:16:06.833887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:06.833858 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:16:06.834275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:06.833968 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:16:09.211537 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:09.211500 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:16:11.052850 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:11.052817 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7751512c_6473_467a_8fad_4fb8bf588636.slice/crio-10e30126f59ce08ed79beeb35c6400f4f4b5a6576d716ce51adf5e1440f83568\": RecentStats: unable to find data in memory cache]" Oct 30 15:16:23.771797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:23.771768 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2"] Oct 30 15:16:23.772279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:23.771984 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" containerID="cri-o://4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc" gracePeriod=30 Oct 30 15:16:23.785931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:23.785905 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww"] Oct 30 15:16:23.786112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:23.786093 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" containerID="cri-o://886f42ec21774dc585796d7bb193ba7ce16c25c26d7b72e250f42e2b2134867e" gracePeriod=30 Oct 30 15:16:23.791677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:23.791651 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn"] Oct 30 15:16:23.791935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:23.791909 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" containerID="cri-o://c0d4f7a0175551227072147df4da726e1ed8bf0cbe9f729905fd1a98a8563e88" gracePeriod=30 Oct 30 15:16:29.117611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.109891 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.120469 6554 generic.go:356] "Generic (PLEG): container finished" podID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerID="c0d4f7a0175551227072147df4da726e1ed8bf0cbe9f729905fd1a98a8563e88" exitCode=0 Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.120540 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" event={"ID":"7e7b09e4-9eda-460a-816a-5686eebcbbd5","Type":"ContainerDied","Data":"c0d4f7a0175551227072147df4da726e1ed8bf0cbe9f729905fd1a98a8563e88"} Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.122172 6554 generic.go:356] "Generic (PLEG): container finished" podID="e5af2bfc-e804-4826-9488-61959cfff44e" containerID="4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc" exitCode=0 Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.122233 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.122241 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" event={"ID":"e5af2bfc-e804-4826-9488-61959cfff44e","Type":"ContainerDied","Data":"4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc"} Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.122291 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2" event={"ID":"e5af2bfc-e804-4826-9488-61959cfff44e","Type":"ContainerDied","Data":"20da19bececbcde089907223002978a69bb3163363b0a97209aeee3c5e5bd24e"} Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.122323 6554 scope.go:122] "RemoveContainer" containerID="4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc" Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.124113 6554 generic.go:356] "Generic (PLEG): container finished" podID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerID="886f42ec21774dc585796d7bb193ba7ce16c25c26d7b72e250f42e2b2134867e" exitCode=0 Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.124188 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" event={"ID":"a5d9982e-1bda-4194-9c76-20258adc90b9","Type":"ContainerDied","Data":"886f42ec21774dc585796d7bb193ba7ce16c25c26d7b72e250f42e2b2134867e"} Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.134547 6554 scope.go:122] "RemoveContainer" containerID="4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc" Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:29.134840 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc\": container with ID starting with 4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc not found: ID does not exist" containerID="4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc" Oct 30 15:16:29.135396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.134916 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc"} err="failed to get container status \"4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc\": rpc error: code = NotFound desc = could not find container \"4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc\": container with ID starting with 4d9ee00ffe14d32aebc8a8708d85430b0178e9bd3c1a3d6d402dc25b502a2edc not found: ID does not exist" Oct 30 15:16:29.158910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.158892 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:16:29.165499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.165480 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:16:29.204938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.204904 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.205097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.204947 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.205097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205034 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.205097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205067 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.205275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205094 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.205275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205122 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.205540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205512 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:29.205634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205574 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.205634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205599 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.205765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205701 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.205765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205748 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.205882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205856 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.205939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.205890 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.206052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206024 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206059 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.206179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206074 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:29.206179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206156 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206138 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.206343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206177 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:29.206343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206223 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.206343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206264 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.206343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206310 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.206343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206311 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206384 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206394 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206513 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206671 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.206824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206783 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.206941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206827 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.206941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206864 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.206941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206896 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.206941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206923 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.207180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.206973 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket\") pod \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\" (UID: \"7e7b09e4-9eda-460a-816a-5686eebcbbd5\") " Oct 30 15:16:29.207180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207040 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.207180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207103 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.207180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207109 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.207180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207124 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.207465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207196 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k\") pod \"a5d9982e-1bda-4194-9c76-20258adc90b9\" (UID: \"a5d9982e-1bda-4194-9c76-20258adc90b9\") " Oct 30 15:16:29.207465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207245 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data\") pod \"e5af2bfc-e804-4826-9488-61959cfff44e\" (UID: \"e5af2bfc-e804-4826-9488-61959cfff44e\") " Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207869 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.207930 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208099 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208229 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208255 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208271 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208285 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208300 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5af2bfc-e804-4826-9488-61959cfff44e-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208314 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208327 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5d9982e-1bda-4194-9c76-20258adc90b9-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208340 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208367 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208380 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208392 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208406 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208419 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.208794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.208432 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.209438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.209406 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.209665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.209637 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.209761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.209686 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.210159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.210132 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:29.210249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.210203 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:29.210324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.210290 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "kube-api-access-n29rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.210699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.210670 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.211281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.211255 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:29.211874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.211848 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:29.211960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.211922 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token" pod "e5af2bfc-e804-4826-9488-61959cfff44e" (UID: "e5af2bfc-e804-4826-9488-61959cfff44e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.212582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.212558 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k" pod "a5d9982e-1bda-4194-9c76-20258adc90b9" (UID: "a5d9982e-1bda-4194-9c76-20258adc90b9"). InnerVolumeSpecName "kube-api-access-bpq9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.212962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.212944 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z" pod "7e7b09e4-9eda-460a-816a-5686eebcbbd5" (UID: "7e7b09e4-9eda-460a-816a-5686eebcbbd5"). InnerVolumeSpecName "kube-api-access-8sf2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:29.309886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309806 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.309886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309841 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpq9k\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-kube-api-access-bpq9k\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.309886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309855 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.309886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309868 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.309886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309879 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309890 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309913 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5af2bfc-e804-4826-9488-61959cfff44e-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309925 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29rn\" (UniqueName: \"kubernetes.io/projected/e5af2bfc-e804-4826-9488-61959cfff44e-kube-api-access-n29rn\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309938 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5d9982e-1bda-4194-9c76-20258adc90b9-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309950 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sf2z\" (UniqueName: \"kubernetes.io/projected/7e7b09e4-9eda-460a-816a-5686eebcbbd5-kube-api-access-8sf2z\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309962 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309974 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7b09e4-9eda-460a-816a-5686eebcbbd5-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.310174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.309986 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5af2bfc-e804-4826-9488-61959cfff44e-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:29.477437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.477403 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2"] Oct 30 15:16:29.480997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:29.480972 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-l99j2"] Oct 30 15:16:30.132717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.132635 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" Oct 30 15:16:30.133178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.132630 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn" event={"ID":"7e7b09e4-9eda-460a-816a-5686eebcbbd5","Type":"ContainerDied","Data":"d8ebff62621b54cdad14f48410d2ce1ddd46a7e1cfb59ee1080ad870fa566f87"} Oct 30 15:16:30.133178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.132794 6554 scope.go:122] "RemoveContainer" containerID="c0d4f7a0175551227072147df4da726e1ed8bf0cbe9f729905fd1a98a8563e88" Oct 30 15:16:30.135609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.135589 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" Oct 30 15:16:30.135734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.135583 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww" event={"ID":"a5d9982e-1bda-4194-9c76-20258adc90b9","Type":"ContainerDied","Data":"ffa4a07a83537fb582421a5005a402a6cc64b59a3ead47d40b0bf98dcd4cd71c"} Oct 30 15:16:30.144223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.144195 6554 scope.go:122] "RemoveContainer" containerID="886f42ec21774dc585796d7bb193ba7ce16c25c26d7b72e250f42e2b2134867e" Oct 30 15:16:30.159168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.159146 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" path="/var/lib/kubelet/pods/e5af2bfc-e804-4826-9488-61959cfff44e/volumes" Oct 30 15:16:30.192547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.192529 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww"] Oct 30 15:16:30.205595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.205576 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-k9tww"] Oct 30 15:16:30.237181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.237162 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn"] Oct 30 15:16:30.240881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:30.240862 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2cfhn"] Oct 30 15:16:32.157359 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:32.157314 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" path="/var/lib/kubelet/pods/7e7b09e4-9eda-460a-816a-5686eebcbbd5/volumes" Oct 30 15:16:32.157843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:32.157826 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" path="/var/lib/kubelet/pods/a5d9982e-1bda-4194-9c76-20258adc90b9/volumes" Oct 30 15:16:38.021503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:38.021453 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c"] Oct 30 15:16:38.030603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:38.030564 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1b7814-wf87c"] Oct 30 15:16:38.166529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:38.166494 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8322d9f-3fe5-478b-a966-aa713bdc06e2" path="/var/lib/kubelet/pods/e8322d9f-3fe5-478b-a966-aa713bdc06e2/volumes" Oct 30 15:16:38.853113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:38.853085 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k"] Oct 30 15:16:38.905794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:38.905761 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck"] Oct 30 15:16:39.204110 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:39.204071 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:16:43.853941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:43.853906 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" podUID="9c1e1202-396d-46fe-b896-68a4b71d85b9" containerName="payload-processing" containerID="cri-o://8c385fb76c0e9647fc8c4d73616103eba1f6e957ce74c8110c37a7704ee238b4" gracePeriod=25 Oct 30 15:16:43.906746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:43.906714 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" podUID="ee72e0a8-acc8-42fd-8351-de68b60cec87" containerName="payload-pre-processing" containerID="cri-o://4d63448871399cf58fa2240e409fbada04c30dc83eb4e7b980e63cc31fbf60e6" gracePeriod=25 Oct 30 15:16:44.315369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.315117 6554 generic.go:356] "Generic (PLEG): container finished" podID="ee72e0a8-acc8-42fd-8351-de68b60cec87" containerID="4d63448871399cf58fa2240e409fbada04c30dc83eb4e7b980e63cc31fbf60e6" exitCode=0 Oct 30 15:16:44.315897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.315858 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" event={"ID":"ee72e0a8-acc8-42fd-8351-de68b60cec87","Type":"ContainerDied","Data":"4d63448871399cf58fa2240e409fbada04c30dc83eb4e7b980e63cc31fbf60e6"} Oct 30 15:16:44.318423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.318259 6554 generic.go:356] "Generic (PLEG): container finished" podID="9c1e1202-396d-46fe-b896-68a4b71d85b9" containerID="8c385fb76c0e9647fc8c4d73616103eba1f6e957ce74c8110c37a7704ee238b4" exitCode=0 Oct 30 15:16:44.318423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.318370 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" event={"ID":"9c1e1202-396d-46fe-b896-68a4b71d85b9","Type":"ContainerDied","Data":"8c385fb76c0e9647fc8c4d73616103eba1f6e957ce74c8110c37a7704ee238b4"} Oct 30 15:16:44.418103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.418074 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:16:44.421512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.421490 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:16:44.482482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.482449 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j\" (UniqueName: \"kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j\") pod \"9c1e1202-396d-46fe-b896-68a4b71d85b9\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " Oct 30 15:16:44.482660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.482596 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume\") pod \"ee72e0a8-acc8-42fd-8351-de68b60cec87\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " Oct 30 15:16:44.482660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.482645 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume\") pod \"9c1e1202-396d-46fe-b896-68a4b71d85b9\" (UID: \"9c1e1202-396d-46fe-b896-68a4b71d85b9\") " Oct 30 15:16:44.482786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.482733 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr\" (UniqueName: \"kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr\") pod \"ee72e0a8-acc8-42fd-8351-de68b60cec87\" (UID: \"ee72e0a8-acc8-42fd-8351-de68b60cec87\") " Oct 30 15:16:44.483014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.482986 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume" pod "ee72e0a8-acc8-42fd-8351-de68b60cec87" (UID: "ee72e0a8-acc8-42fd-8351-de68b60cec87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:44.483151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.483072 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume" pod "9c1e1202-396d-46fe-b896-68a4b71d85b9" (UID: "9c1e1202-396d-46fe-b896-68a4b71d85b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:44.483577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.483552 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee72e0a8-acc8-42fd-8351-de68b60cec87-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:44.483711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.483582 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c1e1202-396d-46fe-b896-68a4b71d85b9-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:44.484912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.484888 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j" pod "9c1e1202-396d-46fe-b896-68a4b71d85b9" (UID: "9c1e1202-396d-46fe-b896-68a4b71d85b9"). InnerVolumeSpecName "kube-api-access-dhx8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:44.485017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.484976 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr" pod "ee72e0a8-acc8-42fd-8351-de68b60cec87" (UID: "ee72e0a8-acc8-42fd-8351-de68b60cec87"). InnerVolumeSpecName "kube-api-access-cg9hr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:44.585939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.585613 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg9hr\" (UniqueName: \"kubernetes.io/projected/ee72e0a8-acc8-42fd-8351-de68b60cec87-kube-api-access-cg9hr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:44.585939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:44.585653 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhx8j\" (UniqueName: \"kubernetes.io/projected/9c1e1202-396d-46fe-b896-68a4b71d85b9-kube-api-access-dhx8j\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:45.326940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.326910 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" Oct 30 15:16:45.326940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.326918 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k" event={"ID":"9c1e1202-396d-46fe-b896-68a4b71d85b9","Type":"ContainerDied","Data":"1d7c20f9da32d4fead5afc7232d2f7a32db393dc261383d14f20de061aa53dc4"} Oct 30 15:16:45.327494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.326974 6554 scope.go:122] "RemoveContainer" containerID="8c385fb76c0e9647fc8c4d73616103eba1f6e957ce74c8110c37a7704ee238b4" Oct 30 15:16:45.329669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.329557 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" Oct 30 15:16:45.329669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.329596 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck" event={"ID":"ee72e0a8-acc8-42fd-8351-de68b60cec87","Type":"ContainerDied","Data":"3fe5be37989caed90f58acfb1070407230cd16afaf4f4f5f58663ed13fb00a1b"} Oct 30 15:16:45.345065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.345037 6554 scope.go:122] "RemoveContainer" containerID="4d63448871399cf58fa2240e409fbada04c30dc83eb4e7b980e63cc31fbf60e6" Oct 30 15:16:45.384309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.384287 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck"] Oct 30 15:16:45.397329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.397298 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-751ab3-58bd55d77f-4kwck"] Oct 30 15:16:45.425711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.425689 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k"] Oct 30 15:16:45.429373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.429336 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-751ab3-c74447484-82n2k"] Oct 30 15:16:45.988654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.988617 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b"] Oct 30 15:16:45.988948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:45.988919 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" containerID="cri-o://8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960" gracePeriod=30 Oct 30 15:16:46.160767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:46.160725 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1e1202-396d-46fe-b896-68a4b71d85b9" path="/var/lib/kubelet/pods/9c1e1202-396d-46fe-b896-68a4b71d85b9/volumes" Oct 30 15:16:46.161267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:46.161245 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee72e0a8-acc8-42fd-8351-de68b60cec87" path="/var/lib/kubelet/pods/ee72e0a8-acc8-42fd-8351-de68b60cec87/volumes" Oct 30 15:16:51.301441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.301416 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:16:51.376502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.376469 6554 generic.go:356] "Generic (PLEG): container finished" podID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerID="8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960" exitCode=0 Oct 30 15:16:51.376660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.376517 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" Oct 30 15:16:51.376660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.376545 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" event={"ID":"b468a794-0400-4e15-947a-4f45c2b7bb8f","Type":"ContainerDied","Data":"8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960"} Oct 30 15:16:51.376660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.376607 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b" event={"ID":"b468a794-0400-4e15-947a-4f45c2b7bb8f","Type":"ContainerDied","Data":"4f273c0abf64c4150f4578012e888e2b7dfec23a26ac4aff3f69c98df745d43c"} Oct 30 15:16:51.376660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.376632 6554 scope.go:122] "RemoveContainer" containerID="8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960" Oct 30 15:16:51.395284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.395264 6554 scope.go:122] "RemoveContainer" containerID="735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9" Oct 30 15:16:51.407776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.407752 6554 scope.go:122] "RemoveContainer" containerID="8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960" Oct 30 15:16:51.408074 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:51.408046 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960\": container with ID starting with 8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960 not found: ID does not exist" containerID="8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960" Oct 30 15:16:51.408182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.408085 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960"} err="failed to get container status \"8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960\": rpc error: code = NotFound desc = could not find container \"8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960\": container with ID starting with 8f47c50bb7bea7ac59f9e168bf7f641326379f696f046768fd2a53da58ccc960 not found: ID does not exist" Oct 30 15:16:51.408182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.408108 6554 scope.go:122] "RemoveContainer" containerID="735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9" Oct 30 15:16:51.408420 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:16:51.408395 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9\": container with ID starting with 735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9 not found: ID does not exist" containerID="735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9" Oct 30 15:16:51.408499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.408428 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9"} err="failed to get container status \"735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9\": rpc error: code = NotFound desc = could not find container \"735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9\": container with ID starting with 735c4fe6c9f0991c372057d78e7ab0b48cd5b2ad26f3e5f227fcf0d475e8e0f9 not found: ID does not exist" Oct 30 15:16:51.416889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.416863 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.416899 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.416939 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417059 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417109 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417219 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:51.417292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417249 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417324 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417448 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:51.417579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417465 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417535 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:51.417691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417635 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo\") pod \"b468a794-0400-4e15-947a-4f45c2b7bb8f\" (UID: \"b468a794-0400-4e15-947a-4f45c2b7bb8f\") " Oct 30 15:16:51.417745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.417720 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:16:51.418128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.418100 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:51.418403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.418384 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.418519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.418408 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.418519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.418422 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.418519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.418435 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.418519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.418448 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b468a794-0400-4e15-947a-4f45c2b7bb8f-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.419974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.419950 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:51.420154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.420128 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:16:51.420258 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.420242 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:16:51.420337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.420324 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv" pod "b468a794-0400-4e15-947a-4f45c2b7bb8f" (UID: "b468a794-0400-4e15-947a-4f45c2b7bb8f"). InnerVolumeSpecName "kube-api-access-rbvpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:16:51.519758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.519695 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.519758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.519718 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.519758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.519726 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.519758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.519738 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbvpv\" (UniqueName: \"kubernetes.io/projected/b468a794-0400-4e15-947a-4f45c2b7bb8f-kube-api-access-rbvpv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:16:51.727327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.727304 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b"] Oct 30 15:16:51.740070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:51.740050 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1b7814-openshift-default-865458f8dc-d5j5b"] Oct 30 15:16:52.156773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:52.156742 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" path="/var/lib/kubelet/pods/b468a794-0400-4e15-947a-4f45c2b7bb8f/volumes" Oct 30 15:16:56.543218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:56.543185 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9"] Oct 30 15:16:56.543692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:56.543397 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" containerID="cri-o://ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08" gracePeriod=30 Oct 30 15:16:57.025633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:16:57.025603 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t"] Oct 30 15:17:00.958313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:00.958255 6554 patch_prober.go:29] interesting pod/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:17:00.958837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:00.958342 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:17:01.335769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.335667 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78"] Oct 30 15:17:01.855169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.855111 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:17:01.951940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.951908 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.951996 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952054 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952079 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952195 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952251 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952283 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952332 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952438 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo\") pod \"722a97e4-2863-49b5-a75c-f51b189279f5\" (UID: \"722a97e4-2863-49b5-a75c-f51b189279f5\") " Oct 30 15:17:01.952945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952812 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:01.952945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.952884 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:01.953118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953093 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:01.953336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953280 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:01.953505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953449 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/722a97e4-2863-49b5-a75c-f51b189279f5-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:01.953505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953471 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:01.953505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953488 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:01.953505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953503 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:01.954006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.953979 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:01.954916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.954890 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:01.955015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.954952 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "kube-api-access-p6qpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:01.955761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.955732 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:17:01.956117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:01.956085 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token" pod "722a97e4-2863-49b5-a75c-f51b189279f5" (UID: "722a97e4-2863-49b5-a75c-f51b189279f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.028441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.027326 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" podUID="bae0ba8e-c9d6-4931-813d-05c25f2398b3" containerName="payload-processing" containerID="cri-o://c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0" gracePeriod=25 Oct 30 15:17:02.058884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.058730 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.058884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.058763 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/722a97e4-2863-49b5-a75c-f51b189279f5-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.058884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.058780 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qpj\" (UniqueName: \"kubernetes.io/projected/722a97e4-2863-49b5-a75c-f51b189279f5-kube-api-access-p6qpj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.058884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.058795 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.058884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.058811 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/722a97e4-2863-49b5-a75c-f51b189279f5-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.323677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.323636 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:17:02.364512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.364473 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6\" (UniqueName: \"kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6\") pod \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " Oct 30 15:17:02.364657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.364625 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume\") pod \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\" (UID: \"bae0ba8e-c9d6-4931-813d-05c25f2398b3\") " Oct 30 15:17:02.364965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.364940 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume" pod "bae0ba8e-c9d6-4931-813d-05c25f2398b3" (UID: "bae0ba8e-c9d6-4931-813d-05c25f2398b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:02.365182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.365165 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae0ba8e-c9d6-4931-813d-05c25f2398b3-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.366701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.366678 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6" pod "bae0ba8e-c9d6-4931-813d-05c25f2398b3" (UID: "bae0ba8e-c9d6-4931-813d-05c25f2398b3"). InnerVolumeSpecName "kube-api-access-6krn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:02.466920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.466831 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6krn6\" (UniqueName: \"kubernetes.io/projected/bae0ba8e-c9d6-4931-813d-05c25f2398b3-kube-api-access-6krn6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:02.467923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.467892 6554 generic.go:356] "Generic (PLEG): container finished" podID="722a97e4-2863-49b5-a75c-f51b189279f5" containerID="ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08" exitCode=0 Oct 30 15:17:02.468069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.468028 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" event={"ID":"722a97e4-2863-49b5-a75c-f51b189279f5","Type":"ContainerDied","Data":"ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08"} Oct 30 15:17:02.468142 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.468082 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" event={"ID":"722a97e4-2863-49b5-a75c-f51b189279f5","Type":"ContainerDied","Data":"3c7da6db6c57ff00a10f781f1891731c554b2995e2abdb9c6241e532da8aafc4"} Oct 30 15:17:02.468142 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.468115 6554 scope.go:122] "RemoveContainer" containerID="ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08" Oct 30 15:17:02.468297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.468282 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9" Oct 30 15:17:02.471866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.471835 6554 generic.go:356] "Generic (PLEG): container finished" podID="bae0ba8e-c9d6-4931-813d-05c25f2398b3" containerID="c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0" exitCode=0 Oct 30 15:17:02.471979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.471959 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" Oct 30 15:17:02.476510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.473976 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" event={"ID":"bae0ba8e-c9d6-4931-813d-05c25f2398b3","Type":"ContainerDied","Data":"c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0"} Oct 30 15:17:02.476510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.474509 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t" event={"ID":"bae0ba8e-c9d6-4931-813d-05c25f2398b3","Type":"ContainerDied","Data":"31a83ad7bae25072ce822388add507b4a92337a09cc70546bdb70fdade61081b"} Oct 30 15:17:02.489554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.489529 6554 scope.go:122] "RemoveContainer" containerID="ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08" Oct 30 15:17:02.491149 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:02.491119 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08\": container with ID starting with ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08 not found: ID does not exist" containerID="ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08" Oct 30 15:17:02.491252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.491160 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08"} err="failed to get container status \"ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08\": rpc error: code = NotFound desc = could not find container \"ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08\": container with ID starting with ebe5cb11a34e0d8f463514c4c2e2500a914a64cdd55ab2edca12de7411d9de08 not found: ID does not exist" Oct 30 15:17:02.491252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.491190 6554 scope.go:122] "RemoveContainer" containerID="c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0" Oct 30 15:17:02.510937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.510913 6554 scope.go:122] "RemoveContainer" containerID="c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0" Oct 30 15:17:02.511242 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:02.511215 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0\": container with ID starting with c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0 not found: ID does not exist" containerID="c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0" Oct 30 15:17:02.511329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.511254 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0"} err="failed to get container status \"c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0\": rpc error: code = NotFound desc = could not find container \"c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0\": container with ID starting with c35914adab02ce770d2104e164ce0d342585f96012c6bdaa54cbdd50d1a154b0 not found: ID does not exist" Oct 30 15:17:02.533272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.533249 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9"] Oct 30 15:17:02.539717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.539696 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-klxk9"] Oct 30 15:17:02.607773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.606568 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t"] Oct 30 15:17:02.607773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:02.607704 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-47bcfe-56d4484666-hhg8t"] Oct 30 15:17:04.156823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:04.156778 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" path="/var/lib/kubelet/pods/722a97e4-2863-49b5-a75c-f51b189279f5/volumes" Oct 30 15:17:04.157422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:04.157397 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae0ba8e-c9d6-4931-813d-05c25f2398b3" path="/var/lib/kubelet/pods/bae0ba8e-c9d6-4931-813d-05c25f2398b3/volumes" Oct 30 15:17:04.575631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:04.575546 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr"] Oct 30 15:17:04.575786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:04.575761 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" podUID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" containerName="maas-api" containerID="cri-o://3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414" gracePeriod=30 Oct 30 15:17:05.138862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.138834 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:17:05.215202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.214285 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877\" (UniqueName: \"kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877\") pod \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " Oct 30 15:17:05.215202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.214395 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\") pod \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " Oct 30 15:17:05.215202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.214501 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca\") pod \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " Oct 30 15:17:05.215202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.214585 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs\") pod \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\" (UID: \"e78c1b17-0ced-4359-a4a6-6f89a5ed098c\") " Oct 30 15:17:05.215202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.214941 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca" pod "e78c1b17-0ced-4359-a4a6-6f89a5ed098c" (UID: "e78c1b17-0ced-4359-a4a6-6f89a5ed098c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:05.223376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.222642 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877" pod "e78c1b17-0ced-4359-a4a6-6f89a5ed098c" (UID: "e78c1b17-0ced-4359-a4a6-6f89a5ed098c"). InnerVolumeSpecName "kube-api-access-hj877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:05.223376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.222696 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls" pod "e78c1b17-0ced-4359-a4a6-6f89a5ed098c" (UID: "e78c1b17-0ced-4359-a4a6-6f89a5ed098c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:17:05.223781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.223757 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs" pod "e78c1b17-0ced-4359-a4a6-6f89a5ed098c" (UID: "e78c1b17-0ced-4359-a4a6-6f89a5ed098c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:17:05.227251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.226495 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:05.327622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.327581 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj877\" (UniqueName: \"kubernetes.io/projected/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-kube-api-access-hj877\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:05.327622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.327613 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:05.327622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.327622 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e78c1b17-0ced-4359-a4a6-6f89a5ed098c-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:05.511680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.511591 6554 generic.go:356] "Generic (PLEG): container finished" podID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" containerID="3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414" exitCode=0 Oct 30 15:17:05.511680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.511656 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" Oct 30 15:17:05.512026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.511684 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" event={"ID":"e78c1b17-0ced-4359-a4a6-6f89a5ed098c","Type":"ContainerDied","Data":"3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414"} Oct 30 15:17:05.512026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.511764 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" event={"ID":"e78c1b17-0ced-4359-a4a6-6f89a5ed098c","Type":"ContainerDied","Data":"3fdfff13c0b87a8fc753db6afe6dbb1ab21d7d53e28b5599f787093bb8006577"} Oct 30 15:17:05.512026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.511792 6554 scope.go:122] "RemoveContainer" containerID="3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414" Oct 30 15:17:05.529554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.526059 6554 scope.go:122] "RemoveContainer" containerID="3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414" Oct 30 15:17:05.529554 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:05.526423 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414\": container with ID starting with 3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414 not found: ID does not exist" containerID="3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414" Oct 30 15:17:05.529554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.526477 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414"} err="failed to get container status \"3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414\": rpc error: code = NotFound desc = could not find container \"3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414\": container with ID starting with 3a343b421f72eb5aa73547c6a648f48896c0964d9b18d9acac5b9353c95c3414 not found: ID does not exist" Oct 30 15:17:05.579178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.579148 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr"] Oct 30 15:17:05.582851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:05.582823 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr"] Oct 30 15:17:06.156525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:06.156489 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" path="/var/lib/kubelet/pods/e78c1b17-0ced-4359-a4a6-6f89a5ed098c/volumes" Oct 30 15:17:06.336117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:06.336079 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerName="payload-processing" containerID="cri-o://3f3a8566dbf30e13d35d3af1de3f5982d91eef7bcbac8dfb59510bd1377527a9" gracePeriod=25 Oct 30 15:17:07.985938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:07.985901 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9254b0-84d7f47b8f-wzmqr" podUID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.123:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:17:09.201600 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:09.201559 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:17:12.848592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:12.848548 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s"] Oct 30 15:17:12.849373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:12.849320 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" containerID="cri-o://b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd" gracePeriod=30 Oct 30 15:17:13.643369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:13.643325 6554 patch_prober.go:29] interesting pod/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:17:13.643549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:13.643402 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:17:14.633654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:14.633621 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:17:16.230823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.230790 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77"] Oct 30 15:17:16.231576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231559 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ee72e0a8-acc8-42fd-8351-de68b60cec87" containerName="payload-pre-processing" Oct 30 15:17:16.231630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231577 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee72e0a8-acc8-42fd-8351-de68b60cec87" containerName="payload-pre-processing" Oct 30 15:17:16.231630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231609 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" Oct 30 15:17:16.231630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231614 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" Oct 30 15:17:16.231630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231626 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" Oct 30 15:17:16.231630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231631 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231642 6554 cpu_manager.go:414] "containerMap: removing container" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231647 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231661 6554 cpu_manager.go:414] "containerMap: removing container" podUID="bae0ba8e-c9d6-4931-813d-05c25f2398b3" containerName="payload-processing" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231665 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae0ba8e-c9d6-4931-813d-05c25f2398b3" containerName="payload-processing" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231674 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" containerName="maas-api" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231678 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" containerName="maas-api" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231686 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231690 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231697 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9c1e1202-396d-46fe-b896-68a4b71d85b9" containerName="payload-processing" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231702 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1e1202-396d-46fe-b896-68a4b71d85b9" containerName="payload-processing" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231713 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231718 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231728 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" Oct 30 15:17:16.231793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231732 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231825 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231834 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7b09e4-9eda-460a-816a-5686eebcbbd5" containerName="istio-proxy" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231839 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="722a97e4-2863-49b5-a75c-f51b189279f5" containerName="istio-proxy" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231846 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae0ba8e-c9d6-4931-813d-05c25f2398b3" containerName="payload-processing" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231854 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e78c1b17-0ced-4359-a4a6-6f89a5ed098c" containerName="maas-api" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231862 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5d9982e-1bda-4194-9c76-20258adc90b9" containerName="istio-proxy" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231869 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1e1202-396d-46fe-b896-68a4b71d85b9" containerName="payload-processing" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231876 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5af2bfc-e804-4826-9488-61959cfff44e" containerName="istio-proxy" Oct 30 15:17:16.232204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.231883 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee72e0a8-acc8-42fd-8351-de68b60cec87" containerName="payload-pre-processing" Oct 30 15:17:16.235791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.235617 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.252466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.252444 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d84bc6-openshift-default-dockercfg-xc7z7\"" Oct 30 15:17:16.270555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.270527 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77"] Oct 30 15:17:16.370242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370200 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.370690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370631 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.370690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370682 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.370901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370719 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.370901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370790 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.370901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370847 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njl82\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.371073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.370955 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.371073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.371009 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.371182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.371103 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.472615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.472578 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.472833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.472653 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njl82\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.472833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.472756 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.472833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.472810 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.472900 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.472990 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473041 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473076 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473114 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473426 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473496 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473499 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.473775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.473755 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.478062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.478033 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.485994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.485922 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.487486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.486281 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.509227 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.509154 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njl82\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.518281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.518250 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token\") pod \"e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.557843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.557813 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:16.892710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.892684 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77"] Oct 30 15:17:16.894066 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:17:16.894037 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a81a0ea_b425_4207_9003_e3d8c748260d.slice/crio-8e408c7f66bc70ae4d54d29b5628160b53dcd1af14f6ea2b1f36a1c624c1639a WatchSource:0}: Error finding container 8e408c7f66bc70ae4d54d29b5628160b53dcd1af14f6ea2b1f36a1c624c1639a: Status 404 returned error can't find the container with id 8e408c7f66bc70ae4d54d29b5628160b53dcd1af14f6ea2b1f36a1c624c1639a Oct 30 15:17:16.896831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.896795 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:16.896942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.896876 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:16.896942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:16.896922 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:17.623177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:17.623102 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" event={"ID":"6a81a0ea-b425-4207-9003-e3d8c748260d","Type":"ContainerStarted","Data":"d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa"} Oct 30 15:17:17.623177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:17.623155 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" event={"ID":"6a81a0ea-b425-4207-9003-e3d8c748260d","Type":"ContainerStarted","Data":"8e408c7f66bc70ae4d54d29b5628160b53dcd1af14f6ea2b1f36a1c624c1639a"} Oct 30 15:17:17.687802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:17.687750 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podStartSLOduration=1.687737908 podStartE2EDuration="1.687737908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:17.684941836 +0000 UTC m=+1741.643983240" watchObservedRunningTime="2026-10-30 15:17:17.687737908 +0000 UTC m=+1741.646779313" Oct 30 15:17:18.136080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.136060 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:17:18.302371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302312 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302455 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302498 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302555 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302582 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.302675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302610 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302749 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.302824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302752 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302850 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.302976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302950 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.303062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.302995 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket\") pod \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\" (UID: \"c5cf74c6-cd6a-46a3-b89b-1582d122ceae\") " Oct 30 15:17:18.303288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.303239 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.303288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.303276 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:18.303542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.303480 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.304104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.304069 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.304104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.304091 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.304104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.304105 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.304387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.304118 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.304387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.304129 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.305151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.305129 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "kube-api-access-rw98t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:18.305237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.305206 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:17:18.305324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.305308 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:17:18.305424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.305311 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token" pod "c5cf74c6-cd6a-46a3-b89b-1582d122ceae" (UID: "c5cf74c6-cd6a-46a3-b89b-1582d122ceae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:18.405756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.405700 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw98t\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-kube-api-access-rw98t\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.405756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.405723 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.405756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.405734 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.405756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.405743 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5cf74c6-cd6a-46a3-b89b-1582d122ceae-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:18.558949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.558922 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:18.565286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.565267 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:18.632199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.632076 6554 generic.go:356] "Generic (PLEG): container finished" podID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerID="b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd" exitCode=0 Oct 30 15:17:18.632199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.632150 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" Oct 30 15:17:18.632914 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.632889 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" event={"ID":"c5cf74c6-cd6a-46a3-b89b-1582d122ceae","Type":"ContainerDied","Data":"b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd"} Oct 30 15:17:18.633078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.632937 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:18.633078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.632955 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s" event={"ID":"c5cf74c6-cd6a-46a3-b89b-1582d122ceae","Type":"ContainerDied","Data":"c41dee388c194069d59c33083db4fc05664793d53943b0d8703d29df712fa582"} Oct 30 15:17:18.633078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.632974 6554 scope.go:122] "RemoveContainer" containerID="b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd" Oct 30 15:17:18.634039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.633944 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:17:18.643924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.643909 6554 scope.go:122] "RemoveContainer" containerID="b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd" Oct 30 15:17:18.644205 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:18.644182 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd\": container with ID starting with b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd not found: ID does not exist" containerID="b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd" Oct 30 15:17:18.644292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.644210 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd"} err="failed to get container status \"b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd\": rpc error: code = NotFound desc = could not find container \"b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd\": container with ID starting with b38798269c7f27eaf740653917c79b433b76caaff834da802fb89770357912dd not found: ID does not exist" Oct 30 15:17:18.690402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.690330 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s"] Oct 30 15:17:18.694855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:18.694833 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9254b0-openshift-default-c57dbbdff-tsl4s"] Oct 30 15:17:20.156949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:20.156915 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" path="/var/lib/kubelet/pods/c5cf74c6-cd6a-46a3-b89b-1582d122ceae/volumes" Oct 30 15:17:22.067098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.067066 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d"] Oct 30 15:17:22.067850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.067833 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" Oct 30 15:17:22.067909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.067851 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" Oct 30 15:17:22.067967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.067958 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cf74c6-cd6a-46a3-b89b-1582d122ceae" containerName="istio-proxy" Oct 30 15:17:22.068006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.067977 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b468a794-0400-4e15-947a-4f45c2b7bb8f" containerName="istio-proxy" Oct 30 15:17:22.073098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.073067 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.090700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.090673 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d84bc6-dockercfg-fz4bs\"" Oct 30 15:17:22.090795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.090762 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d84bc6\"" Oct 30 15:17:22.092784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.092746 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d"] Oct 30 15:17:22.260019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.259979 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mfmm\" (UniqueName: \"kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm\") pod \"payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.260227 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.260137 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.362048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.361958 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mfmm\" (UniqueName: \"kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm\") pod \"payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.362229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.362051 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.363101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.363078 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.387672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.387642 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mfmm\" (UniqueName: \"kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm\") pod \"payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.686908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.686883 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:22.995206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:22.995133 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d"] Oct 30 15:17:22.997135 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:17:22.997110 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dc6f7c_0a67_48af_9563_1569e8217e12.slice/crio-7552e57f163ccc381c452c6f45002ce1d4257cfa1199e2b999b03470e8e843fe WatchSource:0}: Error finding container 7552e57f163ccc381c452c6f45002ce1d4257cfa1199e2b999b03470e8e843fe: Status 404 returned error can't find the container with id 7552e57f163ccc381c452c6f45002ce1d4257cfa1199e2b999b03470e8e843fe Oct 30 15:17:23.690405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:23.690334 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" event={"ID":"94dc6f7c-0a67-48af-9563-1569e8217e12","Type":"ContainerStarted","Data":"4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e"} Oct 30 15:17:23.690813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:23.690456 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" event={"ID":"94dc6f7c-0a67-48af-9563-1569e8217e12","Type":"ContainerStarted","Data":"7552e57f163ccc381c452c6f45002ce1d4257cfa1199e2b999b03470e8e843fe"} Oct 30 15:17:23.690813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:23.690484 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:23.735842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:23.735798 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" podStartSLOduration=1.735785625 podStartE2EDuration="1.735785625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:23.73443555 +0000 UTC m=+1747.693476952" watchObservedRunningTime="2026-10-30 15:17:23.735785625 +0000 UTC m=+1747.694827029" Oct 30 15:17:23.865989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:23.865952 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h"] Oct 30 15:17:23.890401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:23.890366 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4"] Oct 30 15:17:24.633435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:24.633398 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:17:26.756045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.756016 6554 generic.go:356] "Generic (PLEG): container finished" podID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerID="3f3a8566dbf30e13d35d3af1de3f5982d91eef7bcbac8dfb59510bd1377527a9" exitCode=0 Oct 30 15:17:26.756421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.756091 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" event={"ID":"0cfd7637-5771-49c9-be4d-c068ba5c1247","Type":"ContainerDied","Data":"3f3a8566dbf30e13d35d3af1de3f5982d91eef7bcbac8dfb59510bd1377527a9"} Oct 30 15:17:26.756421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.756145 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" event={"ID":"0cfd7637-5771-49c9-be4d-c068ba5c1247","Type":"ContainerDied","Data":"6b36cdabfdcadc00d6bcb3a9232a45c914c931d78ab3fc3afa6015b62795129d"} Oct 30 15:17:26.756421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.756163 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b36cdabfdcadc00d6bcb3a9232a45c914c931d78ab3fc3afa6015b62795129d" Oct 30 15:17:26.775745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.775724 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:17:26.846986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.846952 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc\" (UniqueName: \"kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc\") pod \"0cfd7637-5771-49c9-be4d-c068ba5c1247\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " Oct 30 15:17:26.847140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.847032 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume\") pod \"0cfd7637-5771-49c9-be4d-c068ba5c1247\" (UID: \"0cfd7637-5771-49c9-be4d-c068ba5c1247\") " Oct 30 15:17:26.848384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.848341 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume" pod "0cfd7637-5771-49c9-be4d-c068ba5c1247" (UID: "0cfd7637-5771-49c9-be4d-c068ba5c1247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:26.850299 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.850271 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc" pod "0cfd7637-5771-49c9-be4d-c068ba5c1247" (UID: "0cfd7637-5771-49c9-be4d-c068ba5c1247"). InnerVolumeSpecName "kube-api-access-xgpdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:26.948861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.948834 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgpdc\" (UniqueName: \"kubernetes.io/projected/0cfd7637-5771-49c9-be4d-c068ba5c1247-kube-api-access-xgpdc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:26.948861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:26.948860 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0cfd7637-5771-49c9-be4d-c068ba5c1247-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:27.799322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:27.790191 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78" Oct 30 15:17:27.849656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:27.849621 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78"] Oct 30 15:17:27.884162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:27.884120 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9254b0-669987dbbd-94t78"] Oct 30 15:17:28.179436 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:28.179338 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" path="/var/lib/kubelet/pods/0cfd7637-5771-49c9-be4d-c068ba5c1247/volumes" Oct 30 15:17:28.875446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:28.875402 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" podUID="3053b5a0-4454-426b-b2c3-2dad176dfe79" containerName="payload-processing" containerID="cri-o://1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9" gracePeriod=25 Oct 30 15:17:28.890988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:28.890954 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerName="payload-pre-processing" containerID="cri-o://838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d" gracePeriod=25 Oct 30 15:17:29.227054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.227024 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:17:29.303098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.303063 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl\" (UniqueName: \"kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl\") pod \"3053b5a0-4454-426b-b2c3-2dad176dfe79\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " Oct 30 15:17:29.303260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.303150 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume\") pod \"3053b5a0-4454-426b-b2c3-2dad176dfe79\" (UID: \"3053b5a0-4454-426b-b2c3-2dad176dfe79\") " Oct 30 15:17:29.303823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.303798 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume" pod "3053b5a0-4454-426b-b2c3-2dad176dfe79" (UID: "3053b5a0-4454-426b-b2c3-2dad176dfe79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:29.305435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.305410 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl" pod "3053b5a0-4454-426b-b2c3-2dad176dfe79" (UID: "3053b5a0-4454-426b-b2c3-2dad176dfe79"). InnerVolumeSpecName "kube-api-access-kkccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:29.404568 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.404538 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkccl\" (UniqueName: \"kubernetes.io/projected/3053b5a0-4454-426b-b2c3-2dad176dfe79-kube-api-access-kkccl\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:29.404568 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.404565 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3053b5a0-4454-426b-b2c3-2dad176dfe79-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:29.807049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.807017 6554 generic.go:356] "Generic (PLEG): container finished" podID="3053b5a0-4454-426b-b2c3-2dad176dfe79" containerID="1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9" exitCode=0 Oct 30 15:17:29.807218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.807070 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" Oct 30 15:17:29.807218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.807098 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" event={"ID":"3053b5a0-4454-426b-b2c3-2dad176dfe79","Type":"ContainerDied","Data":"1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9"} Oct 30 15:17:29.807218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.807153 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h" event={"ID":"3053b5a0-4454-426b-b2c3-2dad176dfe79","Type":"ContainerDied","Data":"16cf8230c23b39d5b52a2901c2bd908cb80d23024705da696785c9f4108dcfba"} Oct 30 15:17:29.807218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.807175 6554 scope.go:122] "RemoveContainer" containerID="1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9" Oct 30 15:17:29.818655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.818638 6554 scope.go:122] "RemoveContainer" containerID="1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9" Oct 30 15:17:29.818923 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:29.818902 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9\": container with ID starting with 1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9 not found: ID does not exist" containerID="1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9" Oct 30 15:17:29.818990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.818929 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9"} err="failed to get container status \"1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9\": rpc error: code = NotFound desc = could not find container \"1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9\": container with ID starting with 1e09d08ece038942ebc4b4859c2a4e298ec282458b6a8148b8e4fcb3129bfae9 not found: ID does not exist" Oct 30 15:17:29.882047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.882019 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h"] Oct 30 15:17:29.885333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:29.885311 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-24fff1-5bc7d8c4fb-lvl7h"] Oct 30 15:17:30.175373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:30.160915 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3053b5a0-4454-426b-b2c3-2dad176dfe79" path="/var/lib/kubelet/pods/3053b5a0-4454-426b-b2c3-2dad176dfe79/volumes" Oct 30 15:17:31.704595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:31.704556 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:17:34.702558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:34.702522 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:17:36.521433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.521398 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb"] Oct 30 15:17:36.522245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.522225 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3053b5a0-4454-426b-b2c3-2dad176dfe79" containerName="payload-processing" Oct 30 15:17:36.522245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.522244 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3053b5a0-4454-426b-b2c3-2dad176dfe79" containerName="payload-processing" Oct 30 15:17:36.522465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.522260 6554 cpu_manager.go:414] "containerMap: removing container" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerName="payload-processing" Oct 30 15:17:36.522465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.522267 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerName="payload-processing" Oct 30 15:17:36.522589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.522473 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3053b5a0-4454-426b-b2c3-2dad176dfe79" containerName="payload-processing" Oct 30 15:17:36.522589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.522485 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cfd7637-5771-49c9-be4d-c068ba5c1247" containerName="payload-processing" Oct 30 15:17:36.525330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.525308 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.542967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.542938 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0a4f54-openshift-default-dockercfg-8tmv4\"" Oct 30 15:17:36.560015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.559982 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb"] Oct 30 15:17:36.646640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.646595 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.647036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.647011 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.647243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.647223 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82hxk\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.647438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.647419 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.647563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.647549 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.647715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.647701 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.648188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.648168 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.648368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.648334 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.648497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.648481 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.750081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750054 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750090 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750134 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750158 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750321 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750476 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750515 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82hxk\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750577 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750591 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750640 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750862 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.750998 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.751149 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.751779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.751490 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.758894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.758857 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.781428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.775771 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.781428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.776466 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.781428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.778089 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82hxk\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk\") pod \"e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.844401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.842332 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:36.861274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.860243 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": read tcp 10.134.0.2:58692->10.134.0.158:15021: read: connection reset by peer" start-of-body= Oct 30 15:17:36.861274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:36.860324 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": read tcp 10.134.0.2:58692->10.134.0.158:15021: read: connection reset by peer" Oct 30 15:17:37.217503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.216415 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb"] Oct 30 15:17:37.220254 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:17:37.220189 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe847216_389a_4225_9f40_42e6a1a8adce.slice/crio-a9f0af090b97db989f832c7e75f41e5f7f507d178329eed11daddf84d669d9e6 WatchSource:0}: Error finding container a9f0af090b97db989f832c7e75f41e5f7f507d178329eed11daddf84d669d9e6: Status 404 returned error can't find the container with id a9f0af090b97db989f832c7e75f41e5f7f507d178329eed11daddf84d669d9e6 Oct 30 15:17:37.224035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.224003 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:37.224112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.224069 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:37.224112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.224098 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:37.891055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.891025 6554 generic.go:356] "Generic (PLEG): container finished" podID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerID="28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747" exitCode=0 Oct 30 15:17:37.891559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.891092 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" event={"ID":"46ed6f78-cca3-4e47-b41b-9f0e2200ce88","Type":"ContainerDied","Data":"28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747"} Oct 30 15:17:37.891735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.891719 6554 scope.go:122] "RemoveContainer" containerID="28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747" Oct 30 15:17:37.893270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.893243 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" event={"ID":"be847216-389a-4225-9f40-42e6a1a8adce","Type":"ContainerStarted","Data":"47adce0412e53237094a9f17560fe9172e51096bd39249480c50f9b6bc075a52"} Oct 30 15:17:37.893403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.893298 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" event={"ID":"be847216-389a-4225-9f40-42e6a1a8adce","Type":"ContainerStarted","Data":"a9f0af090b97db989f832c7e75f41e5f7f507d178329eed11daddf84d669d9e6"} Oct 30 15:17:37.893626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.893596 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:37.893703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.893686 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:37.893752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:37.893738 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:17:38.066604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.066540 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podStartSLOduration=2.066520085 podStartE2EDuration="2.066520085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:38.062549039 +0000 UTC m=+1762.021590446" watchObservedRunningTime="2026-10-30 15:17:38.066520085 +0000 UTC m=+1762.025561491" Oct 30 15:17:38.843511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.843478 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:38.849603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.849574 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:38.902871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.902836 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" event={"ID":"46ed6f78-cca3-4e47-b41b-9f0e2200ce88","Type":"ContainerStarted","Data":"1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18"} Oct 30 15:17:38.903319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.903297 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:38.905041 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.904966 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:17:38.928377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.928338 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:17:38.928501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.928485 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:17:38.928553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:38.928536 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 15:17:39.204729 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:39.204693 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:17:39.928129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:39.928101 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:17:39.928537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:39.928147 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 15:17:40.928427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:40.928388 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:17:40.928900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:40.928440 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 15:17:41.705305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:41.705262 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:17:41.928149 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:41.928115 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:17:41.928335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:41.928174 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 15:17:42.705463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.705426 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8"] Oct 30 15:17:42.740727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.726470 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:42.740727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.728653 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8"] Oct 30 15:17:42.749884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.749853 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0a4f54\"" Oct 30 15:17:42.814755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.814725 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0a4f54-dockercfg-mk5k5\"" Oct 30 15:17:42.853174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.853138 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxstr\" (UniqueName: \"kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr\") pod \"payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:42.853408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.853190 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:42.927874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.927840 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:17:42.928070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.927893 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 15:17:42.979541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.979433 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:42.979846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.979819 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxstr\" (UniqueName: \"kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr\") pod \"payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:42.981288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:42.981134 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:43.006281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.006247 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxstr\" (UniqueName: \"kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr\") pod \"payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:43.116145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.116115 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:43.449536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.449507 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8"] Oct 30 15:17:43.928260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.928228 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 15:17:43.928667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.928279 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 15:17:43.940246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.940221 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" event={"ID":"73c9b2a7-b631-48ad-ab9c-48e9934ae680","Type":"ContainerStarted","Data":"23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f"} Oct 30 15:17:43.940391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.940269 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" event={"ID":"73c9b2a7-b631-48ad-ab9c-48e9934ae680","Type":"ContainerStarted","Data":"71eb94d5dd0ba662ebbe9f9dbed84056792471e20db2322ccb566bf0d69fe1c4"} Oct 30 15:17:43.940445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:43.940382 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:17:44.001599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:44.001544 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" podStartSLOduration=2.00153209 podStartE2EDuration="2.00153209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:17:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:17:43.99875251 +0000 UTC m=+1767.957793915" watchObservedRunningTime="2026-10-30 15:17:44.00153209 +0000 UTC m=+1767.960573543" Oct 30 15:17:44.933010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:44.932972 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:17:44.933528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:44.933470 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:17:44.934251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:44.934226 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:17:49.175787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.175759 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:17:49.264743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.264696 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume\") pod \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " Oct 30 15:17:49.264895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.264876 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch\" (UniqueName: \"kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch\") pod \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\" (UID: \"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea\") " Oct 30 15:17:49.265170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.265142 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume" pod "7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" (UID: "7de2f3cb-6eb4-4028-ac32-f1be0318f8ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:17:49.265559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.265531 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:49.267217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.267187 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch" pod "7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" (UID: "7de2f3cb-6eb4-4028-ac32-f1be0318f8ea"). InnerVolumeSpecName "kube-api-access-h7kch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:17:49.367439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.367399 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7kch\" (UniqueName: \"kubernetes.io/projected/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea-kube-api-access-h7kch\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:17:49.985390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.985345 6554 generic.go:356] "Generic (PLEG): container finished" podID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerID="838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d" exitCode=0 Oct 30 15:17:49.985590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.985411 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" Oct 30 15:17:49.985590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.985448 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" event={"ID":"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea","Type":"ContainerDied","Data":"838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d"} Oct 30 15:17:49.985590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.985537 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4" event={"ID":"7de2f3cb-6eb4-4028-ac32-f1be0318f8ea","Type":"ContainerDied","Data":"623de9ac197a55b3c90d0372567e82c1d945f9274e0e894de59c40e112700d93"} Oct 30 15:17:49.985590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.985574 6554 scope.go:122] "RemoveContainer" containerID="838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d" Oct 30 15:17:49.996491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.996475 6554 scope.go:122] "RemoveContainer" containerID="838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d" Oct 30 15:17:49.996754 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:17:49.996736 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d\": container with ID starting with 838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d not found: ID does not exist" containerID="838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d" Oct 30 15:17:49.996794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:49.996761 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d"} err="failed to get container status \"838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d\": rpc error: code = NotFound desc = could not find container \"838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d\": container with ID starting with 838e9a17feac98686d676b468e725b5790377b27db9297d36092653b88b9673d not found: ID does not exist" Oct 30 15:17:50.045334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:50.045298 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4"] Oct 30 15:17:50.049343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:50.049319 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-24fff1-6574ccbd7-rm2b4"] Oct 30 15:17:50.155425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:50.155398 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" path="/var/lib/kubelet/pods/7de2f3cb-6eb4-4028-ac32-f1be0318f8ea/volumes" Oct 30 15:17:54.952661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:17:54.952633 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:18:05.504640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:05.504431 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s"] Oct 30 15:18:05.505206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:05.504725 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" containerID="cri-o://e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf" gracePeriod=30 Oct 30 15:18:08.586539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:08.586497 6554 patch_prober.go:29] interesting pod/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:18:08.586950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:08.586559 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:18:09.220927 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:18:09.220789 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:18:10.792480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.792452 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:18:10.887450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887421 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.887619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887463 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.887619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887581 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.887619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887610 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.887802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887647 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.887983 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887953 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:10.888074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887985 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:10.888074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.887762 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.888193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888134 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.888267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888193 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.888267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888242 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket\") pod \"87100560-14d1-4906-8d43-90e404d4f607\" (UID: \"87100560-14d1-4906-8d43-90e404d4f607\") " Oct 30 15:18:10.888481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888434 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:10.888628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888565 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:10.888954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888901 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:10.888954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888917 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.888954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888942 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.888954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888956 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.889211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.888970 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.890496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.890450 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:10.890647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.890545 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:18:10.890647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.890623 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "kube-api-access-gt4ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:10.890987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.890962 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo" pod "87100560-14d1-4906-8d43-90e404d4f607" (UID: "87100560-14d1-4906-8d43-90e404d4f607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:18:10.990842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.990817 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87100560-14d1-4906-8d43-90e404d4f607-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.990842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.990842 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt4ml\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-kube-api-access-gt4ml\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.990977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.990852 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87100560-14d1-4906-8d43-90e404d4f607-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.990977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.990863 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87100560-14d1-4906-8d43-90e404d4f607-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:10.990977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:10.990875 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87100560-14d1-4906-8d43-90e404d4f607-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:11.148153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.148123 6554 generic.go:356] "Generic (PLEG): container finished" podID="87100560-14d1-4906-8d43-90e404d4f607" containerID="e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf" exitCode=0 Oct 30 15:18:11.148303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.148186 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" Oct 30 15:18:11.148303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.148205 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" event={"ID":"87100560-14d1-4906-8d43-90e404d4f607","Type":"ContainerDied","Data":"e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf"} Oct 30 15:18:11.148303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.148256 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s" event={"ID":"87100560-14d1-4906-8d43-90e404d4f607","Type":"ContainerDied","Data":"47c2cce055cd2faee27b59a59420a6c6526a295ef2d28584c099209d6e758534"} Oct 30 15:18:11.148303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.148279 6554 scope.go:122] "RemoveContainer" containerID="e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf" Oct 30 15:18:11.161124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.161101 6554 scope.go:122] "RemoveContainer" containerID="e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2" Oct 30 15:18:11.171730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.171707 6554 scope.go:122] "RemoveContainer" containerID="e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf" Oct 30 15:18:11.172057 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:18:11.172034 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf\": container with ID starting with e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf not found: ID does not exist" containerID="e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf" Oct 30 15:18:11.172139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.172067 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf"} err="failed to get container status \"e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf\": rpc error: code = NotFound desc = could not find container \"e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf\": container with ID starting with e187e4cd5484497e16e81642be36f401dc46fcc11bb7f2f1dd5843bb673b9baf not found: ID does not exist" Oct 30 15:18:11.172139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.172090 6554 scope.go:122] "RemoveContainer" containerID="e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2" Oct 30 15:18:11.172405 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:18:11.172382 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2\": container with ID starting with e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2 not found: ID does not exist" containerID="e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2" Oct 30 15:18:11.172479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.172414 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2"} err="failed to get container status \"e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2\": rpc error: code = NotFound desc = could not find container \"e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2\": container with ID starting with e249fe11efc1131c42b64ac083dce3c5b405100651caab8f92aeda62f4f036a2 not found: ID does not exist" Oct 30 15:18:11.240927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.240891 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s"] Oct 30 15:18:11.251734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:11.251704 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-751ab3-openshift-default-7dc8d99f8c-mf57s"] Oct 30 15:18:12.156656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:12.156624 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87100560-14d1-4906-8d43-90e404d4f607" path="/var/lib/kubelet/pods/87100560-14d1-4906-8d43-90e404d4f607/volumes" Oct 30 15:18:14.882116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882083 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq"] Oct 30 15:18:14.882821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882806 6554 cpu_manager.go:414] "containerMap: removing container" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" Oct 30 15:18:14.882821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882821 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" Oct 30 15:18:14.882908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882841 6554 cpu_manager.go:414] "containerMap: removing container" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" Oct 30 15:18:14.882908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882846 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" Oct 30 15:18:14.882908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882874 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerName="payload-pre-processing" Oct 30 15:18:14.882908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882881 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerName="payload-pre-processing" Oct 30 15:18:14.883071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.882995 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" Oct 30 15:18:14.883071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.883003 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="87100560-14d1-4906-8d43-90e404d4f607" containerName="istio-proxy" Oct 30 15:18:14.883071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.883014 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de2f3cb-6eb4-4028-ac32-f1be0318f8ea" containerName="payload-pre-processing" Oct 30 15:18:14.888186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.888164 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:14.905433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.905408 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq"] Oct 30 15:18:14.915553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.915515 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a9275a\"" Oct 30 15:18:14.915677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.915548 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a9275a-dockercfg-p7wrn\"" Oct 30 15:18:14.961399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.956994 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbtk2\" (UniqueName: \"kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2\") pod \"payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:14.961399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:14.957076 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:15.060403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.059597 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbtk2\" (UniqueName: \"kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2\") pod \"payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:15.060403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.059672 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:15.067393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.060727 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:15.067393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.061069 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d"] Oct 30 15:18:15.068518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.068496 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d"] Oct 30 15:18:15.068691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.068586 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.093228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.093199 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbtk2\" (UniqueName: \"kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2\") pod \"payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:15.160669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.160638 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7t6\" (UniqueName: \"kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6\") pod \"payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.160819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.160686 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.202882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.202859 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:15.263305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.263270 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7t6\" (UniqueName: \"kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6\") pod \"payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.263413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.263341 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.264301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.264274 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.291766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.289831 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7t6\" (UniqueName: \"kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6\") pod \"payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.411615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.411557 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:15.513786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.513752 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq"] Oct 30 15:18:15.515274 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:18:15.515249 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb4d2c0_31af_4ce0_b10c_ce7c8a33321e.slice/crio-a1b603cd614c732beaaa72f289d5ba803d0253d4ec05b7d4a9244628a19f1930 WatchSource:0}: Error finding container a1b603cd614c732beaaa72f289d5ba803d0253d4ec05b7d4a9244628a19f1930: Status 404 returned error can't find the container with id a1b603cd614c732beaaa72f289d5ba803d0253d4ec05b7d4a9244628a19f1930 Oct 30 15:18:15.720193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:15.720159 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d"] Oct 30 15:18:15.720651 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:18:15.720625 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8a4ed0_e17a_437a_9eba_91356b584128.slice/crio-1348e35f8e20172c2a0839c9096a8ac6b4bafbf824842b418a9b6988516e0769 WatchSource:0}: Error finding container 1348e35f8e20172c2a0839c9096a8ac6b4bafbf824842b418a9b6988516e0769: Status 404 returned error can't find the container with id 1348e35f8e20172c2a0839c9096a8ac6b4bafbf824842b418a9b6988516e0769 Oct 30 15:18:16.186114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.186084 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" event={"ID":"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e","Type":"ContainerStarted","Data":"02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c"} Oct 30 15:18:16.186548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.186132 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" event={"ID":"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e","Type":"ContainerStarted","Data":"a1b603cd614c732beaaa72f289d5ba803d0253d4ec05b7d4a9244628a19f1930"} Oct 30 15:18:16.186548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.186178 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:16.187719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.187694 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" event={"ID":"5b8a4ed0-e17a-437a-9eba-91356b584128","Type":"ContainerStarted","Data":"7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d"} Oct 30 15:18:16.187836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.187738 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" event={"ID":"5b8a4ed0-e17a-437a-9eba-91356b584128","Type":"ContainerStarted","Data":"1348e35f8e20172c2a0839c9096a8ac6b4bafbf824842b418a9b6988516e0769"} Oct 30 15:18:16.187836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.187760 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:16.239482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.239434 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" podStartSLOduration=2.239421227 podStartE2EDuration="2.239421227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:16.237340577 +0000 UTC m=+1800.196381982" watchObservedRunningTime="2026-10-30 15:18:16.239421227 +0000 UTC m=+1800.198462628" Oct 30 15:18:16.283926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.283869 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" podStartSLOduration=2.283852217 podStartE2EDuration="2.283852217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:16.281834026 +0000 UTC m=+1800.240875431" watchObservedRunningTime="2026-10-30 15:18:16.283852217 +0000 UTC m=+1800.242893620" Oct 30 15:18:16.910787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:16.910746 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn"] Oct 30 15:18:17.286808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:17.286732 6554 scope.go:122] "RemoveContainer" containerID="3f3a8566dbf30e13d35d3af1de3f5982d91eef7bcbac8dfb59510bd1377527a9" Oct 30 15:18:21.911827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:21.911771 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" podUID="88fd47f8-b0f8-4485-bb2d-d10db66652a7" containerName="payload-pre-processing" containerID="cri-o://22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d" gracePeriod=25 Oct 30 15:18:22.195628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.195604 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:18:22.250436 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.250394 6554 generic.go:356] "Generic (PLEG): container finished" podID="88fd47f8-b0f8-4485-bb2d-d10db66652a7" containerID="22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d" exitCode=0 Oct 30 15:18:22.250610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.250448 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" event={"ID":"88fd47f8-b0f8-4485-bb2d-d10db66652a7","Type":"ContainerDied","Data":"22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d"} Oct 30 15:18:22.250610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.250495 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" event={"ID":"88fd47f8-b0f8-4485-bb2d-d10db66652a7","Type":"ContainerDied","Data":"8465cc31df79fa5a968f967b2751f99690e8ebeffd5fbc2380607d019f6b05be"} Oct 30 15:18:22.250610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.250524 6554 scope.go:122] "RemoveContainer" containerID="22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d" Oct 30 15:18:22.250794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.250684 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn" Oct 30 15:18:22.266148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.266115 6554 scope.go:122] "RemoveContainer" containerID="22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d" Oct 30 15:18:22.266439 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:18:22.266421 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d\": container with ID starting with 22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d not found: ID does not exist" containerID="22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d" Oct 30 15:18:22.266488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.266447 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d"} err="failed to get container status \"22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d\": rpc error: code = NotFound desc = could not find container \"22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d\": container with ID starting with 22bd739da90c10c1db3b3d3ab17a52a6e94fe45dca19e0be263e1bc24ba8f93d not found: ID does not exist" Oct 30 15:18:22.359293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.359270 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6\" (UniqueName: \"kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6\") pod \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " Oct 30 15:18:22.359543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.359524 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume\") pod \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\" (UID: \"88fd47f8-b0f8-4485-bb2d-d10db66652a7\") " Oct 30 15:18:22.359876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.359850 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume" pod "88fd47f8-b0f8-4485-bb2d-d10db66652a7" (UID: "88fd47f8-b0f8-4485-bb2d-d10db66652a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:22.360048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.360028 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88fd47f8-b0f8-4485-bb2d-d10db66652a7-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.361406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.361379 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6" pod "88fd47f8-b0f8-4485-bb2d-d10db66652a7" (UID: "88fd47f8-b0f8-4485-bb2d-d10db66652a7"). InnerVolumeSpecName "kube-api-access-8mkc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:22.425764 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.425696 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk"] Oct 30 15:18:22.426016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.425920 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" podUID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" containerName="maas-api" containerID="cri-o://a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41" gracePeriod=30 Oct 30 15:18:22.462414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.462382 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mkc6\" (UniqueName: \"kubernetes.io/projected/88fd47f8-b0f8-4485-bb2d-d10db66652a7-kube-api-access-8mkc6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.572336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.572305 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" podUID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.141:8443/health\": dial tcp 10.134.0.141:8443: connect: connection refused" Oct 30 15:18:22.603305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.603270 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn"] Oct 30 15:18:22.607313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.607289 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-496acb-77ff89cc9f-lhgqn"] Oct 30 15:18:22.734067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.731498 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:18:22.867863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.867834 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b\" (UniqueName: \"kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b\") pod \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " Oct 30 15:18:22.868035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.867900 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\") pod \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " Oct 30 15:18:22.868035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.867971 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca\") pod \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " Oct 30 15:18:22.868116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.868109 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs\") pod \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\" (UID: \"b88e57bc-a2e9-494a-a7fd-924b957fef1c\") " Oct 30 15:18:22.868339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.868311 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca" pod "b88e57bc-a2e9-494a-a7fd-924b957fef1c" (UID: "b88e57bc-a2e9-494a-a7fd-924b957fef1c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:18:22.868682 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.868660 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b88e57bc-a2e9-494a-a7fd-924b957fef1c-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.870108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.870088 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls" pod "b88e57bc-a2e9-494a-a7fd-924b957fef1c" (UID: "b88e57bc-a2e9-494a-a7fd-924b957fef1c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:18:22.870194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.870161 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs" pod "b88e57bc-a2e9-494a-a7fd-924b957fef1c" (UID: "b88e57bc-a2e9-494a-a7fd-924b957fef1c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:18:22.870194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.870164 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b" pod "b88e57bc-a2e9-494a-a7fd-924b957fef1c" (UID: "b88e57bc-a2e9-494a-a7fd-924b957fef1c"). InnerVolumeSpecName "kube-api-access-jmc6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:18:22.970530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.970431 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.970530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.970470 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b88e57bc-a2e9-494a-a7fd-924b957fef1c-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:22.970530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:22.970485 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmc6b\" (UniqueName: \"kubernetes.io/projected/b88e57bc-a2e9-494a-a7fd-924b957fef1c-kube-api-access-jmc6b\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:18:23.260136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.260053 6554 generic.go:356] "Generic (PLEG): container finished" podID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" containerID="a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41" exitCode=0 Oct 30 15:18:23.260136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.260105 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" Oct 30 15:18:23.260407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.260132 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" event={"ID":"b88e57bc-a2e9-494a-a7fd-924b957fef1c","Type":"ContainerDied","Data":"a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41"} Oct 30 15:18:23.260407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.260190 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk" event={"ID":"b88e57bc-a2e9-494a-a7fd-924b957fef1c","Type":"ContainerDied","Data":"6a5db9a39b2d2d1d781efa004dd21afd4c8503202575d6a7172c47a08fb76ab5"} Oct 30 15:18:23.260407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.260219 6554 scope.go:122] "RemoveContainer" containerID="a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41" Oct 30 15:18:23.270916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.270893 6554 scope.go:122] "RemoveContainer" containerID="a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41" Oct 30 15:18:23.271187 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:18:23.271163 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41\": container with ID starting with a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41 not found: ID does not exist" containerID="a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41" Oct 30 15:18:23.271283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.271195 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41"} err="failed to get container status \"a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41\": rpc error: code = NotFound desc = could not find container \"a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41\": container with ID starting with a424a3c0a35c1f623e73b85cb68b87b5e42a4a63dd5af7e52d7687c7e5c34a41 not found: ID does not exist" Oct 30 15:18:23.323952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.323926 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk"] Oct 30 15:18:23.326993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:23.326973 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-47bcfe-5448f67744-pwntk"] Oct 30 15:18:24.156004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:24.155971 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88fd47f8-b0f8-4485-bb2d-d10db66652a7" path="/var/lib/kubelet/pods/88fd47f8-b0f8-4485-bb2d-d10db66652a7/volumes" Oct 30 15:18:24.156440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:24.156424 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" path="/var/lib/kubelet/pods/b88e57bc-a2e9-494a-a7fd-924b957fef1c/volumes" Oct 30 15:18:26.805818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.805745 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg"] Oct 30 15:18:26.806507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.806492 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" containerName="maas-api" Oct 30 15:18:26.806564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.806508 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" containerName="maas-api" Oct 30 15:18:26.806564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.806524 6554 cpu_manager.go:414] "containerMap: removing container" podUID="88fd47f8-b0f8-4485-bb2d-d10db66652a7" containerName="payload-pre-processing" Oct 30 15:18:26.806564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.806530 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88fd47f8-b0f8-4485-bb2d-d10db66652a7" containerName="payload-pre-processing" Oct 30 15:18:26.806687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.806633 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b88e57bc-a2e9-494a-a7fd-924b957fef1c" containerName="maas-api" Oct 30 15:18:26.806687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.806643 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="88fd47f8-b0f8-4485-bb2d-d10db66652a7" containerName="payload-pre-processing" Oct 30 15:18:26.811646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.811625 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:26.834445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.834420 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg"] Oct 30 15:18:26.843578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.843554 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0a4f54-serving-cert\"" Oct 30 15:18:26.919400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.919370 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:26.919558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.919491 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:26.919558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.919527 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:26.919684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:26.919661 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4g5r\" (UniqueName: \"kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.020859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.020827 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.020998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.020976 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.021038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.021007 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.021121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.021104 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4g5r\" (UniqueName: \"kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.021518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.021487 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.023455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.023435 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.023538 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.023517 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.047133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.047098 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4g5r\" (UniqueName: \"kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r\") pod \"maas-api-e2e-inf-b-0a4f54-7759485978-727pg\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.126126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.126046 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:27.202216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.201411 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:18:27.204147 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.203017 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:18:27.450254 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:18:27.450226 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006ec327_c900_4c9b_92ab_185d124e6377.slice/crio-144ce494fdad1b4c0170f6d17bae9f424939b404988f051ac317e1c9f3c772ae WatchSource:0}: Error finding container 144ce494fdad1b4c0170f6d17bae9f424939b404988f051ac317e1c9f3c772ae: Status 404 returned error can't find the container with id 144ce494fdad1b4c0170f6d17bae9f424939b404988f051ac317e1c9f3c772ae Oct 30 15:18:27.450406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.450274 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg"] Oct 30 15:18:27.451612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:27.451597 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:18:28.318344 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:28.318306 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" event={"ID":"006ec327-c900-4c9b-92ab-185d124e6377","Type":"ContainerStarted","Data":"d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681"} Oct 30 15:18:28.318777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:28.318378 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" event={"ID":"006ec327-c900-4c9b-92ab-185d124e6377","Type":"ContainerStarted","Data":"144ce494fdad1b4c0170f6d17bae9f424939b404988f051ac317e1c9f3c772ae"} Oct 30 15:18:28.318777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:28.318418 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:28.367530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:28.367477 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" podStartSLOduration=1.943185567 podStartE2EDuration="2.367463622s" totalImagesPullingTime="424.278055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:28.363478466 +0000 UTC m=+1812.322519894" watchObservedRunningTime="2026-10-30 15:18:28.367463622 +0000 UTC m=+1812.326505051" Oct 30 15:18:32.655169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.655131 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk"] Oct 30 15:18:32.660315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.660288 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.677387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.677362 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3acdc3-openshift-default-dockercfg-6vp4g\"" Oct 30 15:18:32.694736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.694709 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk"] Oct 30 15:18:32.811380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811337 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811470 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9nt\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811500 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811611 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811656 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811725 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811826 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811813 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811847 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.811898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.811870 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913412 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9nt\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913478 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913593 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913654 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913690 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913776 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913815 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913845 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.913917 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.914969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.914742 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.919423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.915289 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.919423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.918011 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.919423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.918248 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.919423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.919080 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.925641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.922398 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.934175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.934137 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.968369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.965107 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.978505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.977664 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9nt\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt\") pod \"e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:32.981165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:32.981137 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:33.326038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:33.326009 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk"] Oct 30 15:18:33.326917 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:18:33.326879 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7887332a_4ea2_4d7e_a89e_aab2ffe4a6fc.slice/crio-b9833a3ae97c1dd1f9afc6ad735afc20ada91bb31cb953647becb0a8c2654244 WatchSource:0}: Error finding container b9833a3ae97c1dd1f9afc6ad735afc20ada91bb31cb953647becb0a8c2654244: Status 404 returned error can't find the container with id b9833a3ae97c1dd1f9afc6ad735afc20ada91bb31cb953647becb0a8c2654244 Oct 30 15:18:33.329702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:33.329665 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:18:33.329813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:33.329757 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:18:33.329813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:33.329807 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:18:33.359604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:33.359580 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" event={"ID":"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc","Type":"ContainerStarted","Data":"b9833a3ae97c1dd1f9afc6ad735afc20ada91bb31cb953647becb0a8c2654244"} Oct 30 15:18:34.330343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:34.330315 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:18:34.367919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:34.367890 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" event={"ID":"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc","Type":"ContainerStarted","Data":"814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6"} Oct 30 15:18:34.425739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:34.425677 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podStartSLOduration=2.425657981 podStartE2EDuration="2.425657981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:18:34.423950447 +0000 UTC m=+1818.382991854" watchObservedRunningTime="2026-10-30 15:18:34.425657981 +0000 UTC m=+1818.384699388" Oct 30 15:18:34.982509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:34.982477 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:34.987814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:34.987791 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:35.375163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:35.375089 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:35.375987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:35.375954 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:18:39.300175 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:18:39.300136 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:18:48.477933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.477902 6554 generic.go:356] "Generic (PLEG): container finished" podID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerID="d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa" exitCode=0 Oct 30 15:18:48.478438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.477981 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" event={"ID":"6a81a0ea-b425-4207-9003-e3d8c748260d","Type":"ContainerDied","Data":"d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa"} Oct 30 15:18:48.478438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.478372 6554 scope.go:122] "RemoveContainer" containerID="d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa" Oct 30 15:18:48.479941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.479911 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:18:48.480027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.479982 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:18:48.480027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.480018 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:18:48.633831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:48.633800 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:18:49.526244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:49.526202 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" event={"ID":"6a81a0ea-b425-4207-9003-e3d8c748260d","Type":"ContainerStarted","Data":"4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb"} Oct 30 15:18:49.560631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:49.560604 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:18:49.572308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:49.572274 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 15:18:49.572945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:49.572585 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 15:18:50.577373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:50.576558 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 15:18:50.577373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:50.576610 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 15:18:51.561671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:51.561634 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 15:18:51.561949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:51.561919 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 15:18:52.578406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:52.575667 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 15:18:52.578406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:52.575725 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 15:18:53.573587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:53.573528 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 15:18:53.578371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:53.573865 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 15:18:54.558956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:54.558920 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" start-of-body= Oct 30 15:18:54.559507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:54.558984 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.161:15021/healthz/ready\": dial tcp 10.134.0.161:15021: connect: connection refused" Oct 30 15:18:55.564135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:55.564101 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:18:55.564622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:55.564604 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:18:55.565205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:18:55.565183 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:19:09.202102 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:19:09.202058 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:19:19.024698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.024664 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7"] Oct 30 15:19:19.030572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.030551 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.051390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.051365 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7"] Oct 30 15:19:19.073448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.073422 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-3acdc3-serving-cert\"" Oct 30 15:19:19.096500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.096034 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf47g\" (UniqueName: \"kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.096500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.096248 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.096500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.096283 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.096500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.096367 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.197162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.197132 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf47g\" (UniqueName: \"kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.197311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.197245 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.197311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.197271 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.197466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.197323 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.197466 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:19:19.197407 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-serving-cert: secret "maas-api-e2e-inf-a-3acdc3-serving-cert" not found Oct 30 15:19:19.197587 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:19:19.197475 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls podName:16503d59-114c-408a-9ea1-a60f4f6e663a nodeName:}" failed. No retries permitted until 2026-10-30 15:19:19.697459535 +0000 UTC m=+1863.656500931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls") pod "maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" (UID: "16503d59-114c-408a-9ea1-a60f4f6e663a") : secret "maas-api-e2e-inf-a-3acdc3-serving-cert" not found Oct 30 15:19:19.198027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.198006 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.199765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.199742 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.222636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.222615 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf47g\" (UniqueName: \"kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.709597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.709549 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.711915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.711887 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\") pod \"maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:19.944807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:19.944774 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:20.294144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:20.294113 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7"] Oct 30 15:19:20.295364 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:19:20.295323 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16503d59_114c_408a_9ea1_a60f4f6e663a.slice/crio-db2429c2cf41d8f01d3cc59879dcb64879bb03978148efffb35157784a210229 WatchSource:0}: Error finding container db2429c2cf41d8f01d3cc59879dcb64879bb03978148efffb35157784a210229: Status 404 returned error can't find the container with id db2429c2cf41d8f01d3cc59879dcb64879bb03978148efffb35157784a210229 Oct 30 15:19:20.824825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:20.824772 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" event={"ID":"16503d59-114c-408a-9ea1-a60f4f6e663a","Type":"ContainerStarted","Data":"aa8ef84da091d339500fbbd3f739797d6be54bb8f285efc1bec4bb0abefeb7fa"} Oct 30 15:19:20.824982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:20.824853 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:20.824982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:20.824879 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" event={"ID":"16503d59-114c-408a-9ea1-a60f4f6e663a","Type":"ContainerStarted","Data":"db2429c2cf41d8f01d3cc59879dcb64879bb03978148efffb35157784a210229"} Oct 30 15:19:20.879874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:20.879742 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" podStartSLOduration=2.490085121 podStartE2EDuration="2.879721617s" totalImagesPullingTime="389.636496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:20.871729657 +0000 UTC m=+1864.830771077" watchObservedRunningTime="2026-10-30 15:19:20.879721617 +0000 UTC m=+1864.838763024" Oct 30 15:19:23.040402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.040365 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p"] Oct 30 15:19:23.045219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.045197 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.081486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.081456 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p"] Oct 30 15:19:23.145986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.145953 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a9275a-openshift-default-dockercfg-pkhdl\"" Oct 30 15:19:23.186015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.185984 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186027 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jzp\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186104 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186159 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186209 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186303 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186404 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186427 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.186572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.186555 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288254 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288527 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288564 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288703 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288824 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288864 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jzp\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.288917 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.289011 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.289063 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.289991 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.290221 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.293340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.291041 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.296434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.296410 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.298819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.298792 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.307306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.306711 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.307964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.307933 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.314303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.314280 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.317726 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.317701 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jzp\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp\") pod \"e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.372015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.371988 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:23.703684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.703657 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p"] Oct 30 15:19:23.704769 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:19:23.704739 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55618dab_3ed3_40bc_8f0e_68e4bc6893a0.slice/crio-c537a9b37b52d4de942602c95e0ce071f5bd21081a7a4f814272bd10e2819ec4 WatchSource:0}: Error finding container c537a9b37b52d4de942602c95e0ce071f5bd21081a7a4f814272bd10e2819ec4: Status 404 returned error can't find the container with id c537a9b37b52d4de942602c95e0ce071f5bd21081a7a4f814272bd10e2819ec4 Oct 30 15:19:23.708071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.708033 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:19:23.708152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.708103 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:19:23.708152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.708131 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:19:23.852499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.851335 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" event={"ID":"55618dab-3ed3-40bc-8f0e-68e4bc6893a0","Type":"ContainerStarted","Data":"cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0"} Oct 30 15:19:23.852499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.851416 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" event={"ID":"55618dab-3ed3-40bc-8f0e-68e4bc6893a0","Type":"ContainerStarted","Data":"c537a9b37b52d4de942602c95e0ce071f5bd21081a7a4f814272bd10e2819ec4"} Oct 30 15:19:23.904338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:23.904279 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podStartSLOduration=0.90426059 podStartE2EDuration="904.26059ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:23.901392945 +0000 UTC m=+1867.860434353" watchObservedRunningTime="2026-10-30 15:19:23.90426059 +0000 UTC m=+1867.863301997" Oct 30 15:19:24.372223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:24.372184 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:24.374048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:24.374018 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:19:24.374222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:24.374067 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 15:19:24.944547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:24.944464 6554 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:19:24.944547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:24.944535 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:19:25.374811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:25.374722 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:19:25.375433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:25.375399 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 15:19:26.378482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:26.378442 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:19:26.379029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:26.378503 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 15:19:26.837290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:26.837261 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:19:27.378934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:27.378473 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:19:27.378934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:27.378537 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 15:19:28.378769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.378668 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:19:28.378769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.378722 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 15:19:28.909066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.909035 6554 generic.go:356] "Generic (PLEG): container finished" podID="be847216-389a-4225-9f40-42e6a1a8adce" containerID="47adce0412e53237094a9f17560fe9172e51096bd39249480c50f9b6bc075a52" exitCode=0 Oct 30 15:19:28.909234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.909091 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" event={"ID":"be847216-389a-4225-9f40-42e6a1a8adce","Type":"ContainerDied","Data":"47adce0412e53237094a9f17560fe9172e51096bd39249480c50f9b6bc075a52"} Oct 30 15:19:28.909517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.909501 6554 scope.go:122] "RemoveContainer" containerID="47adce0412e53237094a9f17560fe9172e51096bd39249480c50f9b6bc075a52" Oct 30 15:19:28.911321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.911288 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:19:28.911417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.911400 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:19:28.911459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:28.911447 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:19:29.382423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:29.372635 6554 patch_prober.go:29] interesting pod/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body= Oct 30 15:19:29.382423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:29.372692 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" Oct 30 15:19:29.918156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:29.918119 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" event={"ID":"be847216-389a-4225-9f40-42e6a1a8adce","Type":"ContainerStarted","Data":"da2954ecbe0666b67560ddd3da459c86aced9a0febed9e7f5e51abbb936060bc"} Oct 30 15:19:30.401657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:30.401624 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:30.402980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:30.402958 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:30.403582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:30.403562 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:19:30.843864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:30.843782 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:19:30.844019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:30.843962 6554 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 15:19:30.844064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:30.844010 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 15:19:31.843307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:31.843278 6554 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 15:19:31.843708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:31.843332 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 15:19:32.845559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:32.843811 6554 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 15:19:32.845559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:32.843877 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 15:19:33.843574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:33.843542 6554 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 15:19:33.843772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:33.843595 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 15:19:34.843935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:34.843899 6554 patch_prober.go:29] interesting pod/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body= Oct 30 15:19:34.844407 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:34.843948 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" Oct 30 15:19:35.848317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:35.848287 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:19:35.848781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:35.848764 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:19:35.849405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:35.849380 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:19:39.202160 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:19:39.202124 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:19:42.146215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:42.146183 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw"] Oct 30 15:19:42.146738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:42.146405 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" containerID="cri-o://d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b" gracePeriod=30 Oct 30 15:19:42.316802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:42.316769 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl"] Oct 30 15:19:42.317011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:42.316988 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" containerID="cri-o://366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2" gracePeriod=30 Oct 30 15:19:47.321801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.321762 6554 patch_prober.go:29] interesting pod/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 15:19:47.322462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.321822 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 15:19:47.486167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.486132 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:19:47.497814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.497782 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.498465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.498445 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.498646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.498628 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.498850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.498832 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.499012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.498996 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.499151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.499134 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.500543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.499277 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.500543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.499414 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.500543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.499455 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy\") pod \"ce60949b-1edc-45a1-b35a-10b694575297\" (UID: \"ce60949b-1edc-45a1-b35a-10b694575297\") " Oct 30 15:19:47.502438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.498391 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:19:47.502438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.501812 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.502438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.502135 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.502741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.502703 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.503056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.503037 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.513173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.513138 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "kube-api-access-dbvnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.513495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.513460 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.514559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.514537 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.515244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.515220 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo" pod "ce60949b-1edc-45a1-b35a-10b694575297" (UID: "ce60949b-1edc-45a1-b35a-10b694575297"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:19:47.601395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601344 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601395 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce60949b-1edc-45a1-b35a-10b694575297-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601409 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601423 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce60949b-1edc-45a1-b35a-10b694575297-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601435 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601447 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601459 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601473 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvnv\" (UniqueName: \"kubernetes.io/projected/ce60949b-1edc-45a1-b35a-10b694575297-kube-api-access-dbvnv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.601870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.601488 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce60949b-1edc-45a1-b35a-10b694575297-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.617792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.617763 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:19:47.705000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.704902 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.705178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705033 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.705178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705064 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.705178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705096 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.705512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705483 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.705627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705600 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.705830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705808 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:19:47.705912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705893 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.705977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.705913 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.708213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.708175 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.806780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.806747 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.806959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.806789 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.806959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.806828 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.806959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.806858 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.806959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.806907 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data\") pod \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\" (UID: \"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c\") " Oct 30 15:19:47.807513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.807494 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.807513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.807512 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.807718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.807700 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.808238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.808209 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:19:47.809817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.809791 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.809928 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.809906 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:19:47.810116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.810096 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2" pod "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" (UID: "bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c"). InnerVolumeSpecName "kube-api-access-x9nv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:19:47.908554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.908517 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.908554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.908553 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9nv2\" (UniqueName: \"kubernetes.io/projected/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-kube-api-access-x9nv2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.908777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.908566 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.908777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.908581 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:47.908777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:47.908593 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:19:48.109039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.108955 6554 generic.go:356] "Generic (PLEG): container finished" podID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerID="366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2" exitCode=0 Oct 30 15:19:48.109039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.109012 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" Oct 30 15:19:48.109231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.109045 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" event={"ID":"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c","Type":"ContainerDied","Data":"366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2"} Oct 30 15:19:48.109231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.109130 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl" event={"ID":"bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c","Type":"ContainerDied","Data":"ddb8038bc5be1c6a7421f947478f75d8a3121b005bdbc49c8244f8651bac9a84"} Oct 30 15:19:48.109231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.109153 6554 scope.go:122] "RemoveContainer" containerID="366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2" Oct 30 15:19:48.110845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.110817 6554 generic.go:356] "Generic (PLEG): container finished" podID="ce60949b-1edc-45a1-b35a-10b694575297" containerID="d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b" exitCode=0 Oct 30 15:19:48.110985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.110862 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" Oct 30 15:19:48.110985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.110873 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" event={"ID":"ce60949b-1edc-45a1-b35a-10b694575297","Type":"ContainerDied","Data":"d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b"} Oct 30 15:19:48.110985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.110913 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw" event={"ID":"ce60949b-1edc-45a1-b35a-10b694575297","Type":"ContainerDied","Data":"6508ab37c204827080d5c5f532e569473a757fdf84ee6f740dc9906605353669"} Oct 30 15:19:48.120666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.120649 6554 scope.go:122] "RemoveContainer" containerID="366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2" Oct 30 15:19:48.120998 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:19:48.120977 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2\": container with ID starting with 366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2 not found: ID does not exist" containerID="366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2" Oct 30 15:19:48.121083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.121004 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2"} err="failed to get container status \"366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2\": rpc error: code = NotFound desc = could not find container \"366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2\": container with ID starting with 366db021ff9ba6e4583b568c1365a90d6af94b91d13527aa2c0e11d9dfdf74d2 not found: ID does not exist" Oct 30 15:19:48.121083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.121021 6554 scope.go:122] "RemoveContainer" containerID="d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b" Oct 30 15:19:48.130782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.130768 6554 scope.go:122] "RemoveContainer" containerID="d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b" Oct 30 15:19:48.131043 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:19:48.131024 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b\": container with ID starting with d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b not found: ID does not exist" containerID="d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b" Oct 30 15:19:48.131086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.131049 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b"} err="failed to get container status \"d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b\": rpc error: code = NotFound desc = could not find container \"d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b\": container with ID starting with d4fab81b22b50e6393db3a9ce14dab2134ea8ecbd5bb6795915c677d14518c1b not found: ID does not exist" Oct 30 15:19:48.166672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.166652 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl"] Oct 30 15:19:48.180464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.180444 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-f6chl"] Oct 30 15:19:48.214802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.214781 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw"] Oct 30 15:19:48.222976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:48.222951 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-998rw"] Oct 30 15:19:49.188565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.188535 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g"] Oct 30 15:19:49.189263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.189249 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" Oct 30 15:19:49.189303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.189264 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" Oct 30 15:19:49.189303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.189279 6554 cpu_manager.go:414] "containerMap: removing container" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" Oct 30 15:19:49.189303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.189284 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" Oct 30 15:19:49.189443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.189426 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce60949b-1edc-45a1-b35a-10b694575297" containerName="istio-proxy" Oct 30 15:19:49.189443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.189435 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" containerName="istio-proxy" Oct 30 15:19:49.194603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.194587 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.215134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.215111 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-859a8d-dockercfg-6j64h\"" Oct 30 15:19:49.229148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.229123 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-859a8d\"" Oct 30 15:19:49.233010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.232989 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g"] Oct 30 15:19:49.234613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.234558 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.234769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.234748 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twnlp\" (UniqueName: \"kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp\") pod \"payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.251957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.251929 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf"] Oct 30 15:19:49.256478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.256454 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.278614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.278586 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf"] Oct 30 15:19:49.335779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.335742 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.335779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.335801 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.339694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.339666 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.339946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.339912 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twnlp\" (UniqueName: \"kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp\") pod \"payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.340430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.340234 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt4sm\" (UniqueName: \"kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm\") pod \"payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.385370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.385180 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twnlp\" (UniqueName: \"kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp\") pod \"payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.446423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.441875 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt4sm\" (UniqueName: \"kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm\") pod \"payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.446423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.441994 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.446423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.443209 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.467481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.467454 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt4sm\" (UniqueName: \"kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm\") pod \"payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.509019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.508989 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:49.571450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.570776 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:49.823015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.822985 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g"] Oct 30 15:19:49.823298 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:19:49.823277 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a9e8d_c6c2_4089_8250_51dee9f0e38e.slice/crio-27c195418eb1bacf964e3e6a9fafffda3834bc9a6e26c136ba8c4c3dd2da46ad WatchSource:0}: Error finding container 27c195418eb1bacf964e3e6a9fafffda3834bc9a6e26c136ba8c4c3dd2da46ad: Status 404 returned error can't find the container with id 27c195418eb1bacf964e3e6a9fafffda3834bc9a6e26c136ba8c4c3dd2da46ad Oct 30 15:19:49.932655 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:19:49.932625 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dae30e_ec65_427e_80a9_6c72cb470a49.slice/crio-f16c4e932b53c1ef8c7d7b52bb6fd014f590f59d06f78e3fd67cfd2bd69474a5 WatchSource:0}: Error finding container f16c4e932b53c1ef8c7d7b52bb6fd014f590f59d06f78e3fd67cfd2bd69474a5: Status 404 returned error can't find the container with id f16c4e932b53c1ef8c7d7b52bb6fd014f590f59d06f78e3fd67cfd2bd69474a5 Oct 30 15:19:49.932884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:49.932861 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf"] Oct 30 15:19:50.129121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.129033 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" event={"ID":"e5dae30e-ec65-427e-80a9-6c72cb470a49","Type":"ContainerStarted","Data":"3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f"} Oct 30 15:19:50.129121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.129104 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:19:50.129295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.129132 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" event={"ID":"e5dae30e-ec65-427e-80a9-6c72cb470a49","Type":"ContainerStarted","Data":"f16c4e932b53c1ef8c7d7b52bb6fd014f590f59d06f78e3fd67cfd2bd69474a5"} Oct 30 15:19:50.130748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.130721 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" event={"ID":"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e","Type":"ContainerStarted","Data":"531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a"} Oct 30 15:19:50.130873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.130767 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" event={"ID":"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e","Type":"ContainerStarted","Data":"27c195418eb1bacf964e3e6a9fafffda3834bc9a6e26c136ba8c4c3dd2da46ad"} Oct 30 15:19:50.130873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.130791 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:19:50.157888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.157861 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c" path="/var/lib/kubelet/pods/bc7174b8-c9bc-4d5b-939f-f1a1ebf4511c/volumes" Oct 30 15:19:50.158780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.158760 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce60949b-1edc-45a1-b35a-10b694575297" path="/var/lib/kubelet/pods/ce60949b-1edc-45a1-b35a-10b694575297/volumes" Oct 30 15:19:50.174072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.174031 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" podStartSLOduration=1.174020833 podStartE2EDuration="1.174020833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:50.172838282 +0000 UTC m=+1894.131879690" watchObservedRunningTime="2026-10-30 15:19:50.174020833 +0000 UTC m=+1894.133062235" Oct 30 15:19:50.222391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:50.222333 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" podStartSLOduration=1.2223220399999999 podStartE2EDuration="1.22232204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:50.218249637 +0000 UTC m=+1894.177291055" watchObservedRunningTime="2026-10-30 15:19:50.22232204 +0000 UTC m=+1894.181363445" Oct 30 15:19:52.458666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.458629 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws"] Oct 30 15:19:52.464446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.464427 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.484034 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.484009 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws"] Oct 30 15:19:52.492658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.492638 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7c5235-dockercfg-zvhnv\"" Oct 30 15:19:52.492658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.492652 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7c5235\"" Oct 30 15:19:52.535502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.535466 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm"] Oct 30 15:19:52.543196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.543160 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.591399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.589993 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtl7r\" (UniqueName: \"kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r\") pod \"payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.591399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.590117 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kw2w\" (UniqueName: \"kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w\") pod \"payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.591399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.590206 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.591399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.590321 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.660596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.660564 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm"] Oct 30 15:19:52.692373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.692165 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtl7r\" (UniqueName: \"kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r\") pod \"payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.692373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.692287 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kw2w\" (UniqueName: \"kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w\") pod \"payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.692628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.692389 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.692628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.692514 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.693750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.693506 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.694842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.694800 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.719200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.719127 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtl7r\" (UniqueName: \"kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r\") pod \"payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:52.719342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.719328 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kw2w\" (UniqueName: \"kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w\") pod \"payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.779616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.779583 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:52.902455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:52.902405 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:53.094745 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:19:53.094715 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda89b90da_c993_43e2_961e_9add1860eec6.slice/crio-20ad3cac087058c532d486ee9f9bc36f98e3c2514b5fde0477a2c1b85afe60f3 WatchSource:0}: Error finding container 20ad3cac087058c532d486ee9f9bc36f98e3c2514b5fde0477a2c1b85afe60f3: Status 404 returned error can't find the container with id 20ad3cac087058c532d486ee9f9bc36f98e3c2514b5fde0477a2c1b85afe60f3 Oct 30 15:19:53.094890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:53.094875 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws"] Oct 30 15:19:53.155751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:53.155727 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" event={"ID":"a89b90da-c993-43e2-961e-9add1860eec6","Type":"ContainerStarted","Data":"20ad3cac087058c532d486ee9f9bc36f98e3c2514b5fde0477a2c1b85afe60f3"} Oct 30 15:19:53.218743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:53.218706 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm"] Oct 30 15:19:53.224469 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:19:53.224061 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea3ae15e_034a_4ed7_a456_744a1066bde7.slice/crio-92c3aa61c23ca520229898460bef8c58f7a9c5aef2c441e954fd4f5849f9678e WatchSource:0}: Error finding container 92c3aa61c23ca520229898460bef8c58f7a9c5aef2c441e954fd4f5849f9678e: Status 404 returned error can't find the container with id 92c3aa61c23ca520229898460bef8c58f7a9c5aef2c441e954fd4f5849f9678e Oct 30 15:19:54.163903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.163870 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" event={"ID":"a89b90da-c993-43e2-961e-9add1860eec6","Type":"ContainerStarted","Data":"c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd"} Oct 30 15:19:54.164363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.164097 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:19:54.165604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.165576 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" event={"ID":"ea3ae15e-034a-4ed7-a456-744a1066bde7","Type":"ContainerStarted","Data":"c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a"} Oct 30 15:19:54.165712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.165623 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" event={"ID":"ea3ae15e-034a-4ed7-a456-744a1066bde7","Type":"ContainerStarted","Data":"92c3aa61c23ca520229898460bef8c58f7a9c5aef2c441e954fd4f5849f9678e"} Oct 30 15:19:54.165712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.165652 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:19:54.208723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.208680 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" podStartSLOduration=2.208667968 podStartE2EDuration="2.208667968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:54.206345747 +0000 UTC m=+1898.165387153" watchObservedRunningTime="2026-10-30 15:19:54.208667968 +0000 UTC m=+1898.167709368" Oct 30 15:19:54.251447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:19:54.251398 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" podStartSLOduration=2.251387593 podStartE2EDuration="2.251387593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:19:54.249262179 +0000 UTC m=+1898.208303590" watchObservedRunningTime="2026-10-30 15:19:54.251387593 +0000 UTC m=+1898.210428997" Oct 30 15:20:01.142230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:01.142130 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:20:01.142230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:01.142215 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:20:05.181146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:05.181115 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:20:05.181673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:05.181187 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:20:09.202743 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:20:09.202707 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:20:11.386753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:11.386718 6554 generic.go:356] "Generic (PLEG): container finished" podID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerID="814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6" exitCode=0 Oct 30 15:20:11.387185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:11.386790 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" event={"ID":"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc","Type":"ContainerDied","Data":"814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6"} Oct 30 15:20:11.387185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:11.387135 6554 scope.go:122] "RemoveContainer" containerID="814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6" Oct 30 15:20:11.388768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:11.388741 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:20:11.388836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:11.388806 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:20:11.388877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:11.388840 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:20:12.406359 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:12.406302 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" event={"ID":"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc","Type":"ContainerStarted","Data":"87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d"} Oct 30 15:20:12.982670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:12.982630 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:20:12.982853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:12.982832 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:20:12.982909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:12.982887 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 15:20:13.982337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:13.982299 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:20:13.982779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:13.982382 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 15:20:14.982859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:14.982822 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:20:14.983376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:14.982896 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 15:20:15.982265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:15.982231 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:20:15.982504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:15.982299 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 15:20:16.982872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:16.982830 6554 patch_prober.go:29] interesting pod/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body= Oct 30 15:20:16.983290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:16.982897 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" Oct 30 15:20:17.987432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:17.987404 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:20:17.987960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:17.987940 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:20:17.988649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:17.988625 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:20:32.054825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.054790 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s"] Oct 30 15:20:32.061304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.061284 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.093996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.093974 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-859a8d-serving-cert\"" Oct 30 15:20:32.097862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.097839 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s"] Oct 30 15:20:32.122103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.122073 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.122262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.122237 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.122362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.122315 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh588\" (UniqueName: \"kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.122439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.122400 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.224077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.224047 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.224210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.224111 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh588\" (UniqueName: \"kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.224210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.224166 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.224336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.224325 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.225206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.225173 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.226712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.226686 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.226821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.226804 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.248698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.248675 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh588\" (UniqueName: \"kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588\") pod \"maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.375495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.375419 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:32.698207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:32.698178 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s"] Oct 30 15:20:32.698800 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:20:32.698765 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66877f18_0b78_4635_b53d_e5d7fc3e6a62.slice/crio-72c1ae0885abe4a49ae56a1ee90a5e613076931f98c8a7485e93476bdd1eaf61 WatchSource:0}: Error finding container 72c1ae0885abe4a49ae56a1ee90a5e613076931f98c8a7485e93476bdd1eaf61: Status 404 returned error can't find the container with id 72c1ae0885abe4a49ae56a1ee90a5e613076931f98c8a7485e93476bdd1eaf61 Oct 30 15:20:33.605283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:33.605248 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" event={"ID":"66877f18-0b78-4635-b53d-e5d7fc3e6a62","Type":"ContainerStarted","Data":"6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b"} Oct 30 15:20:33.605714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:33.605312 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" event={"ID":"66877f18-0b78-4635-b53d-e5d7fc3e6a62","Type":"ContainerStarted","Data":"72c1ae0885abe4a49ae56a1ee90a5e613076931f98c8a7485e93476bdd1eaf61"} Oct 30 15:20:33.605714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:33.605366 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:33.652698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:33.652634 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" podStartSLOduration=1.253197052 podStartE2EDuration="1.652616167s" totalImagesPullingTime="399.419115ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:20:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:20:33.649806862 +0000 UTC m=+1937.608848268" watchObservedRunningTime="2026-10-30 15:20:33.652616167 +0000 UTC m=+1937.611657576" Oct 30 15:20:37.504791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.504759 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk"] Oct 30 15:20:37.510475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.510456 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.529758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.529731 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7c5235-serving-cert\"" Oct 30 15:20:37.533203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.533178 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk"] Oct 30 15:20:37.615386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.615343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.615633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.615607 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfbr\" (UniqueName: \"kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.615735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.615724 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.615773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.615747 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.717259 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.717227 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.717431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.717420 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfbr\" (UniqueName: \"kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.717507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.717495 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.717576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.717521 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.718102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.718072 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.719999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.719974 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.720113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.719974 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.742264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.742237 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfbr\" (UniqueName: \"kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr\") pod \"maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:37.826468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:37.826398 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:38.158891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:38.158863 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk"] Oct 30 15:20:38.159895 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:20:38.159867 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263d797f_de64_4beb_8fb0_84811b1a22b7.slice/crio-2500dcc3cba6e3b45004ce3e8c7e2acec8c9a8939d752c89e6a896b17fae0f7d WatchSource:0}: Error finding container 2500dcc3cba6e3b45004ce3e8c7e2acec8c9a8939d752c89e6a896b17fae0f7d: Status 404 returned error can't find the container with id 2500dcc3cba6e3b45004ce3e8c7e2acec8c9a8939d752c89e6a896b17fae0f7d Oct 30 15:20:38.654767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:38.654733 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" event={"ID":"263d797f-de64-4beb-8fb0-84811b1a22b7","Type":"ContainerStarted","Data":"f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c"} Oct 30 15:20:38.655167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:38.654790 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" event={"ID":"263d797f-de64-4beb-8fb0-84811b1a22b7","Type":"ContainerStarted","Data":"2500dcc3cba6e3b45004ce3e8c7e2acec8c9a8939d752c89e6a896b17fae0f7d"} Oct 30 15:20:38.655167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:38.654819 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:20:38.704783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:38.704722 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" podStartSLOduration=1.392681985 podStartE2EDuration="1.704709369s" totalImagesPullingTime="312.027384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:20:38.703718273 +0000 UTC m=+1942.662759677" watchObservedRunningTime="2026-10-30 15:20:38.704709369 +0000 UTC m=+1942.663750811" Oct 30 15:20:39.202676 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:20:39.202637 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:20:39.617813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:39.617742 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:20:44.668597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:20:44.668566 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:21:06.042024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.041992 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2"] Oct 30 15:21:06.047868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.047849 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.076539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.076512 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-af2072-serving-cert\"" Oct 30 15:21:06.080917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.080892 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2"] Oct 30 15:21:06.192813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.192780 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgv6v\" (UniqueName: \"kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.192965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.192826 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.192965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.192954 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.193109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.193092 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.294163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.294077 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.294363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.294259 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.294497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.294479 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv6v\" (UniqueName: \"kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.294567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.294515 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.294906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.294873 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.296934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.296907 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.297008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.296967 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.320909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.320882 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv6v\" (UniqueName: \"kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v\") pod \"maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.362655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.362621 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:06.674586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.674554 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2"] Oct 30 15:21:06.675129 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:21:06.675105 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacbbb8e_285b_4e9a_be21_ca29117f6dfc.slice/crio-53213190c5e1db678c6f17ca515ed021b059b86575da6e706f25b44ac7675b79 WatchSource:0}: Error finding container 53213190c5e1db678c6f17ca515ed021b059b86575da6e706f25b44ac7675b79: Status 404 returned error can't find the container with id 53213190c5e1db678c6f17ca515ed021b059b86575da6e706f25b44ac7675b79 Oct 30 15:21:06.875636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:06.875601 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" event={"ID":"dacbbb8e-285b-4e9a-be21-ca29117f6dfc","Type":"ContainerStarted","Data":"53213190c5e1db678c6f17ca515ed021b059b86575da6e706f25b44ac7675b79"} Oct 30 15:21:07.884091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:07.884055 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" event={"ID":"dacbbb8e-285b-4e9a-be21-ca29117f6dfc","Type":"ContainerStarted","Data":"ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85"} Oct 30 15:21:07.884523 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:07.884123 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:07.941994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:07.941566 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" podStartSLOduration=1.633654241 podStartE2EDuration="1.941546711s" totalImagesPullingTime="307.89247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:07.932242952 +0000 UTC m=+1971.891284362" watchObservedRunningTime="2026-10-30 15:21:07.941546711 +0000 UTC m=+1971.900588160" Oct 30 15:21:09.201997 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:21:09.201964 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:21:13.896581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:13.896554 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:21:23.137624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:23.137589 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb"] Oct 30 15:21:23.138148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:23.137823 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" containerID="cri-o://3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3" gracePeriod=30 Oct 30 15:21:27.901308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:27.901229 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h"] Oct 30 15:21:27.906217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:27.906198 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:27.924200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:27.924179 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-416d92-openshift-default-dockercfg-7l9zb\"" Oct 30 15:21:27.944390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:27.944365 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h"] Oct 30 15:21:28.007473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007442 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w7tn\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007490 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007605 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007679 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007801 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007827 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007904 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007935 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.007990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.007957 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109483 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w7tn\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109539 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109568 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109636 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109742 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109770 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109836 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109864 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.109902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.109901 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.110152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.110030 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.110152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.110090 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.110324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.110302 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.110324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.110316 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.110506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.110487 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.112082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.112061 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.112362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.112329 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.133854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.133829 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.134503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.134485 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w7tn\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.219881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.219857 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:28.430296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.430270 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:21:28.515910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.515879 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.515910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.515911 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.515980 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516028 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516073 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516134 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516181 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516233 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.516408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516325 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket\") pod \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\" (UID: \"7c91760f-f50c-4ce7-800d-a8c40cee1ef0\") " Oct 30 15:21:28.517109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516721 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:28.517109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516752 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:28.517109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.516938 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:28.517109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.517081 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:28.517452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.517016 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:21:28.517452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.517189 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.517452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.517224 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.517452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.517240 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.519333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.519305 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:21:28.519333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.519312 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:21:28.519554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.519504 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "kube-api-access-dznqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:21:28.519632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.519613 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy" pod "7c91760f-f50c-4ce7-800d-a8c40cee1ef0" (UID: "7c91760f-f50c-4ce7-800d-a8c40cee1ef0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:21:28.547127 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:21:28.547103 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d982a69_0c73_4b3d_9e1b_388309b69ffd.slice/crio-7accbccb7010982118ce0343a5230f428f49af6aa69ada6bd09b78f57c6d7d52 WatchSource:0}: Error finding container 7accbccb7010982118ce0343a5230f428f49af6aa69ada6bd09b78f57c6d7d52: Status 404 returned error can't find the container with id 7accbccb7010982118ce0343a5230f428f49af6aa69ada6bd09b78f57c6d7d52 Oct 30 15:21:28.547382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.547342 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h"] Oct 30 15:21:28.550525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.550494 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:21:28.550634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.550553 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:21:28.550634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.550581 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:21:28.618948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.618925 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dznqr\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-kube-api-access-dznqr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.619094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.618951 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.619094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.618962 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.619094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.618970 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.619094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.618977 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:28.619094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:28.618985 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c91760f-f50c-4ce7-800d-a8c40cee1ef0-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:21:29.151372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.128504 6554 generic.go:356] "Generic (PLEG): container finished" podID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerID="3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3" exitCode=0 Oct 30 15:21:29.151372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.128680 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" event={"ID":"7c91760f-f50c-4ce7-800d-a8c40cee1ef0","Type":"ContainerDied","Data":"3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3"} Oct 30 15:21:29.151372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.128733 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" event={"ID":"7c91760f-f50c-4ce7-800d-a8c40cee1ef0","Type":"ContainerDied","Data":"32dc8057bc04d97f5bd86326a154d46f7838fbd741ad7af53b6b8b803eae1b3c"} Oct 30 15:21:29.151372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.128765 6554 scope.go:122] "RemoveContainer" containerID="3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3" Oct 30 15:21:29.151372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.128930 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb" Oct 30 15:21:29.179556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.169941 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" event={"ID":"2d982a69-0c73-4b3d-9e1b-388309b69ffd","Type":"ContainerStarted","Data":"7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0"} Oct 30 15:21:29.179556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.170004 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" event={"ID":"2d982a69-0c73-4b3d-9e1b-388309b69ffd","Type":"ContainerStarted","Data":"7accbccb7010982118ce0343a5230f428f49af6aa69ada6bd09b78f57c6d7d52"} Oct 30 15:21:29.210467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.210435 6554 scope.go:122] "RemoveContainer" containerID="3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3" Oct 30 15:21:29.210891 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:21:29.210859 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3\": container with ID starting with 3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3 not found: ID does not exist" containerID="3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3" Oct 30 15:21:29.210987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.210905 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3"} err="failed to get container status \"3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3\": rpc error: code = NotFound desc = could not find container \"3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3\": container with ID starting with 3a4686bf5311c9605dd90dac44c6303681240b7bbbcb46f2c902351064cfd2b3 not found: ID does not exist" Oct 30 15:21:29.221075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.221002 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:29.222924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.222893 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 15:21:29.223141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.222949 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 15:21:29.230387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.229505 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podStartSLOduration=2.229493444 podStartE2EDuration="2.229493444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:21:29.223522377 +0000 UTC m=+1993.182563797" watchObservedRunningTime="2026-10-30 15:21:29.229493444 +0000 UTC m=+1993.188534843" Oct 30 15:21:29.270587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.270552 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb"] Oct 30 15:21:29.283288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:29.283257 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-76473c-openshift-default-7d64f9495-wg7vb"] Oct 30 15:21:30.181814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:30.181750 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" path="/var/lib/kubelet/pods/7c91760f-f50c-4ce7-800d-a8c40cee1ef0/volumes" Oct 30 15:21:30.220456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:30.220429 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 15:21:30.220630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:30.220477 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 15:21:31.220982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:31.220842 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 15:21:31.220982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:31.220902 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 15:21:32.220296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:32.220148 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 15:21:32.220296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:32.220208 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 15:21:33.220647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:33.220610 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 15:21:33.221043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:33.220670 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 15:21:34.220298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:34.220242 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" start-of-body= Oct 30 15:21:34.220494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:34.220361 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.178:15021/healthz/ready\": dial tcp 10.134.0.178:15021: connect: connection refused" Oct 30 15:21:35.469231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:35.469200 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:35.470659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:35.470635 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:35.472059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:21:35.472000 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:21:39.202286 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:21:39.202245 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:22:09.222714 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:22:09.222633 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:22:17.485859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:22:17.485823 6554 scope.go:122] "RemoveContainer" containerID="c717aa023fc0eb84999982a2a2b0eb004a30b46e3312828218a4b5b86a579090" Oct 30 15:22:39.203083 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:22:39.203044 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:23:03.884823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:03.884632 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n"] Oct 30 15:23:03.885450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:03.884907 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" containerID="cri-o://0dc4158959a391525d0aa43805a528cc28f96b54a03cca2aa01e8423f2b83866" gracePeriod=30 Oct 30 15:23:09.144819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.144789 6554 generic.go:356] "Generic (PLEG): container finished" podID="ab402479-669a-4335-8c47-b7bce678d1d8" containerID="0dc4158959a391525d0aa43805a528cc28f96b54a03cca2aa01e8423f2b83866" exitCode=0 Oct 30 15:23:09.145236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.144857 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" event={"ID":"ab402479-669a-4335-8c47-b7bce678d1d8","Type":"ContainerDied","Data":"0dc4158959a391525d0aa43805a528cc28f96b54a03cca2aa01e8423f2b83866"} Oct 30 15:23:09.145236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.144919 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" event={"ID":"ab402479-669a-4335-8c47-b7bce678d1d8","Type":"ContainerDied","Data":"66a0b6ea8b496d73fcf34a3eba1a682e60528476767dc0f94dce8600c704257e"} Oct 30 15:23:09.145236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.144939 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a0b6ea8b496d73fcf34a3eba1a682e60528476767dc0f94dce8600c704257e" Oct 30 15:23:09.145236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.144955 6554 scope.go:122] "RemoveContainer" containerID="d8ab73cdd61ed7866151621d541b96bc3cee4ceec1d201b46b9d4aa1fda759fb" Oct 30 15:23:09.166333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.166314 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:23:09.202397 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:23:09.202360 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:23:09.288145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288110 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.288338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288281 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.288338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288310 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.288488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288392 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.288664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288639 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.288956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288678 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.288956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288787 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.288956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288906 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.289225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288963 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.289225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.288998 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.289225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289029 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo\") pod \"ab402479-669a-4335-8c47-b7bce678d1d8\" (UID: \"ab402479-669a-4335-8c47-b7bce678d1d8\") " Oct 30 15:23:09.289225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289110 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:09.289490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289342 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.289555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289475 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.289853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289830 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.289953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289858 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.289953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289873 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab402479-669a-4335-8c47-b7bce678d1d8-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.289953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289888 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.289953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.289904 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.290748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.290721 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:09.291274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.291245 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "kube-api-access-kppb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:09.291777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.291750 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:23:09.291900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.291881 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy" pod "ab402479-669a-4335-8c47-b7bce678d1d8" (UID: "ab402479-669a-4335-8c47-b7bce678d1d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:09.391028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.391003 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppb2\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-kube-api-access-kppb2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.391028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.391028 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab402479-669a-4335-8c47-b7bce678d1d8-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.391185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.391037 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab402479-669a-4335-8c47-b7bce678d1d8-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:09.391185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:09.391045 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab402479-669a-4335-8c47-b7bce678d1d8-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:10.156330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:10.156301 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n" Oct 30 15:23:10.264617 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:10.264586 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n"] Oct 30 15:23:10.267788 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:10.267766 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a65941-openshift-default-6c47d77999-gms4n"] Oct 30 15:23:12.156912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:12.156874 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" path="/var/lib/kubelet/pods/ab402479-669a-4335-8c47-b7bce678d1d8/volumes" Oct 30 15:23:17.525957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:17.525930 6554 scope.go:122] "RemoveContainer" containerID="0dc4158959a391525d0aa43805a528cc28f96b54a03cca2aa01e8423f2b83866" Oct 30 15:23:25.985480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:25.985444 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5"] Oct 30 15:23:26.023770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:26.023741 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb"] Oct 30 15:23:27.948791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:27.948758 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf"] Oct 30 15:23:27.949213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:27.949022 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" containerID="cri-o://a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e" gracePeriod=30 Oct 30 15:23:30.986862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:30.986525 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" containerName="payload-processing" containerID="cri-o://a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb" gracePeriod=25 Oct 30 15:23:31.026666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:31.026104 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" podUID="c7f03875-b078-4c81-b512-26e085ff6388" containerName="payload-pre-processing" containerID="cri-o://0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5" gracePeriod=25 Oct 30 15:23:32.516496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:32.516460 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" podUID="c7f03875-b078-4c81-b512-26e085ff6388" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:32.516952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:32.516457 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:33.232581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.232524 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:23:33.260236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260206 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260242 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260269 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260329 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260405 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260519 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260545 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:33.260664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260556 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260696 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260758 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data\") pod \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\" (UID: \"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa\") " Oct 30 15:23:33.260992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260849 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:33.260992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.260872 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261127 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261425 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261659 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261679 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261693 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261709 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.262204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.261723 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.263094 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.263067 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:33.263255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.263232 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:23:33.264092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.264066 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:33.265310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.265284 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl" pod "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" (UID: "e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa"). InnerVolumeSpecName "kube-api-access-qhwhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:33.363376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.363325 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.363376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.363378 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhwhl\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-kube-api-access-qhwhl\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.363567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.363390 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.363567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.363399 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:33.405932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.405904 6554 generic.go:356] "Generic (PLEG): container finished" podID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerID="a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e" exitCode=0 Oct 30 15:23:33.406047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.405950 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" Oct 30 15:23:33.406047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.405985 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" event={"ID":"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa","Type":"ContainerDied","Data":"a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e"} Oct 30 15:23:33.406047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.406034 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf" event={"ID":"e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa","Type":"ContainerDied","Data":"28178aa86194fc1d0991f863d79f2bdd1c92dd901ee9ba1377e591c3ba89649a"} Oct 30 15:23:33.406174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.406056 6554 scope.go:122] "RemoveContainer" containerID="a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e" Oct 30 15:23:33.426622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.418729 6554 scope.go:122] "RemoveContainer" containerID="a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e" Oct 30 15:23:33.426622 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:23:33.419171 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e\": container with ID starting with a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e not found: ID does not exist" containerID="a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e" Oct 30 15:23:33.426622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.419207 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e"} err="failed to get container status \"a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e\": rpc error: code = NotFound desc = could not find container \"a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e\": container with ID starting with a29b1f5584b89d9f154be18208aa887d177ec5b5da3fd8b4ca991c2e3128f93e not found: ID does not exist" Oct 30 15:23:33.491974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.491899 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf"] Oct 30 15:23:33.497248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:33.497205 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-vnvgf"] Oct 30 15:23:34.156520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:34.156487 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" path="/var/lib/kubelet/pods/e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa/volumes" Oct 30 15:23:35.594904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:35.594867 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46"] Oct 30 15:23:35.633101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:35.633072 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l"] Oct 30 15:23:39.206376 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:23:39.206317 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:23:40.596081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:40.596045 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerName="payload-processing" containerID="cri-o://60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289" gracePeriod=25 Oct 30 15:23:40.633336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:40.633304 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" podUID="654bc5f1-11ca-40ce-b688-471db30261be" containerName="payload-pre-processing" containerID="cri-o://e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f" gracePeriod=25 Oct 30 15:23:42.516282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:42.516238 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:42.516818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:42.516245 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" podUID="c7f03875-b078-4c81-b512-26e085ff6388" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:42.665604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:42.665570 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl"] Oct 30 15:23:42.665833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:42.665809 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" containerID="cri-o://2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3" gracePeriod=30 Oct 30 15:23:46.831789 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:46.831754 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:46.832202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:46.831810 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" podUID="654bc5f1-11ca-40ce-b688-471db30261be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:47.943476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:47.943422 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:23:48.065824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.065792 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.065846 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.065892 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.065961 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066019 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066103 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066153 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.066399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066253 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.066399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066266 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066343 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066341 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.066620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066442 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.066620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066478 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy\") pod \"8e39c5bd-d7e5-4342-9663-12b1237020f2\" (UID: \"8e39c5bd-d7e5-4342-9663-12b1237020f2\") " Oct 30 15:23:48.066620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.066482 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:48.067245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.067219 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.067329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.067253 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e39c5bd-d7e5-4342-9663-12b1237020f2-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.067329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.067267 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.067329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.067281 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.067329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.067295 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.068235 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.068208 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:23:48.068526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.068503 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:23:48.068861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.068842 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:48.068936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.068867 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649" pod "8e39c5bd-d7e5-4342-9663-12b1237020f2" (UID: "8e39c5bd-d7e5-4342-9663-12b1237020f2"). InnerVolumeSpecName "kube-api-access-fh649". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:48.168243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.168217 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.168243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.168243 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.168403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.168254 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.168403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.168263 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh649\" (UniqueName: \"kubernetes.io/projected/8e39c5bd-d7e5-4342-9663-12b1237020f2-kube-api-access-fh649\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:48.556006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.555970 6554 generic.go:356] "Generic (PLEG): container finished" podID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerID="2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3" exitCode=0 Oct 30 15:23:48.556215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.556026 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" Oct 30 15:23:48.556215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.556078 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" event={"ID":"8e39c5bd-d7e5-4342-9663-12b1237020f2","Type":"ContainerDied","Data":"2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3"} Oct 30 15:23:48.556215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.556150 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl" event={"ID":"8e39c5bd-d7e5-4342-9663-12b1237020f2","Type":"ContainerDied","Data":"ff6e474d835a31c32068d7b6483655d9841bbb564ef66e72d76c8b0b532aa60c"} Oct 30 15:23:48.556215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.556184 6554 scope.go:122] "RemoveContainer" containerID="2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3" Oct 30 15:23:48.567305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.567294 6554 scope.go:122] "RemoveContainer" containerID="2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3" Oct 30 15:23:48.567611 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:23:48.567590 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3\": container with ID starting with 2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3 not found: ID does not exist" containerID="2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3" Oct 30 15:23:48.567692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.567616 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3"} err="failed to get container status \"2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3\": rpc error: code = NotFound desc = could not find container \"2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3\": container with ID starting with 2734a85f64b454b7d3d66a42b1ccd6ce2bdc11cc50825922d3841f1b60d812b3 not found: ID does not exist" Oct 30 15:23:48.613292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.613268 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl"] Oct 30 15:23:48.626868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:48.626845 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-4f5c30-openshift-default-69446885c-2n2fl"] Oct 30 15:23:50.159591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:50.159556 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" path="/var/lib/kubelet/pods/8e39c5bd-d7e5-4342-9663-12b1237020f2/volumes" Oct 30 15:23:51.236783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.236755 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:23:51.351420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.351388 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:23:51.413948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.413918 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume\") pod \"9ba806c4-36f9-4df7-8430-27caa0178955\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " Oct 30 15:23:51.414065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.414046 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k\" (UniqueName: \"kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k\") pod \"9ba806c4-36f9-4df7-8430-27caa0178955\" (UID: \"9ba806c4-36f9-4df7-8430-27caa0178955\") " Oct 30 15:23:51.414372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.414329 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume" pod "9ba806c4-36f9-4df7-8430-27caa0178955" (UID: "9ba806c4-36f9-4df7-8430-27caa0178955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:51.414815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.414783 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba806c4-36f9-4df7-8430-27caa0178955-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:51.416200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.416170 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k" pod "9ba806c4-36f9-4df7-8430-27caa0178955" (UID: "9ba806c4-36f9-4df7-8430-27caa0178955"). InnerVolumeSpecName "kube-api-access-bsc8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:51.515494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.515432 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume\") pod \"c7f03875-b078-4c81-b512-26e085ff6388\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " Oct 30 15:23:51.515640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.515510 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg\" (UniqueName: \"kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg\") pod \"c7f03875-b078-4c81-b512-26e085ff6388\" (UID: \"c7f03875-b078-4c81-b512-26e085ff6388\") " Oct 30 15:23:51.515844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.515820 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume" pod "c7f03875-b078-4c81-b512-26e085ff6388" (UID: "c7f03875-b078-4c81-b512-26e085ff6388"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:23:51.515989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.515973 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsc8k\" (UniqueName: \"kubernetes.io/projected/9ba806c4-36f9-4df7-8430-27caa0178955-kube-api-access-bsc8k\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:51.516045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.515995 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7f03875-b078-4c81-b512-26e085ff6388-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:51.517633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.517614 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg" pod "c7f03875-b078-4c81-b512-26e085ff6388" (UID: "c7f03875-b078-4c81-b512-26e085ff6388"). InnerVolumeSpecName "kube-api-access-j7txg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:23:51.584961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.584933 6554 generic.go:356] "Generic (PLEG): container finished" podID="9ba806c4-36f9-4df7-8430-27caa0178955" containerID="a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb" exitCode=0 Oct 30 15:23:51.585063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.584984 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" Oct 30 15:23:51.585063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.585018 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" event={"ID":"9ba806c4-36f9-4df7-8430-27caa0178955","Type":"ContainerDied","Data":"a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb"} Oct 30 15:23:51.585186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.585095 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5" event={"ID":"9ba806c4-36f9-4df7-8430-27caa0178955","Type":"ContainerDied","Data":"d7678c82c1f953d822bdffa79c3201b61298627ebbfce1eaf4e47d3074c4db92"} Oct 30 15:23:51.585186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.585118 6554 scope.go:122] "RemoveContainer" containerID="a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb" Oct 30 15:23:51.586552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.586527 6554 generic.go:356] "Generic (PLEG): container finished" podID="c7f03875-b078-4c81-b512-26e085ff6388" containerID="0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5" exitCode=0 Oct 30 15:23:51.586694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.586556 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" event={"ID":"c7f03875-b078-4c81-b512-26e085ff6388","Type":"ContainerDied","Data":"0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5"} Oct 30 15:23:51.586694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.586579 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" Oct 30 15:23:51.586694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.586606 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb" event={"ID":"c7f03875-b078-4c81-b512-26e085ff6388","Type":"ContainerDied","Data":"72d58a2a94d60471ef4820d134c8738435526b93d5994f315e0bc8773dd07efe"} Oct 30 15:23:51.597006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.596983 6554 scope.go:122] "RemoveContainer" containerID="a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb" Oct 30 15:23:51.597310 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:23:51.597281 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb\": container with ID starting with a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb not found: ID does not exist" containerID="a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb" Oct 30 15:23:51.597402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.597320 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb"} err="failed to get container status \"a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb\": rpc error: code = NotFound desc = could not find container \"a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb\": container with ID starting with a269d86142d8d6e96ac40eeddb87d00d5bd752aa3c3d33aeddb71a956e6f2bbb not found: ID does not exist" Oct 30 15:23:51.597402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.597344 6554 scope.go:122] "RemoveContainer" containerID="0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5" Oct 30 15:23:51.608118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.608098 6554 scope.go:122] "RemoveContainer" containerID="0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5" Oct 30 15:23:51.608382 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:23:51.608364 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5\": container with ID starting with 0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5 not found: ID does not exist" containerID="0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5" Oct 30 15:23:51.608454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.608392 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5"} err="failed to get container status \"0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5\": rpc error: code = NotFound desc = could not find container \"0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5\": container with ID starting with 0befec70a147f40db1f6d59a5dbb84e814837c3ec6ebacdefc0b6c64e0d54ea5 not found: ID does not exist" Oct 30 15:23:51.617101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.617082 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7txg\" (UniqueName: \"kubernetes.io/projected/c7f03875-b078-4c81-b512-26e085ff6388-kube-api-access-j7txg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:23:51.642822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.642799 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb"] Oct 30 15:23:51.648547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.648523 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-34218d-6758d596b7-h9ggb"] Oct 30 15:23:51.684839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.684812 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5"] Oct 30 15:23:51.688840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:51.688817 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-34218d-6bf977bc96-gjqb5"] Oct 30 15:23:52.156518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:52.156487 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" path="/var/lib/kubelet/pods/9ba806c4-36f9-4df7-8430-27caa0178955/volumes" Oct 30 15:23:52.156901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:52.156888 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f03875-b078-4c81-b512-26e085ff6388" path="/var/lib/kubelet/pods/c7f03875-b078-4c81-b512-26e085ff6388/volumes" Oct 30 15:23:56.165024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:56.164994 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d"] Oct 30 15:23:56.849395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:56.847444 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:23:56.863404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:23:56.853496 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" podUID="654bc5f1-11ca-40ce-b688-471db30261be" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:00.961285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:00.961253 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:24:01.015169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.015146 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:24:01.064417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.064386 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79\" (UniqueName: \"kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79\") pod \"654bc5f1-11ca-40ce-b688-471db30261be\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " Oct 30 15:24:01.064577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.064518 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume\") pod \"654bc5f1-11ca-40ce-b688-471db30261be\" (UID: \"654bc5f1-11ca-40ce-b688-471db30261be\") " Oct 30 15:24:01.064692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.064673 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw\" (UniqueName: \"kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw\") pod \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " Oct 30 15:24:01.064822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.064704 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume\") pod \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\" (UID: \"43bd7df0-55e9-4aea-ad3e-324dd477c83c\") " Oct 30 15:24:01.064936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.064909 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume" pod "654bc5f1-11ca-40ce-b688-471db30261be" (UID: "654bc5f1-11ca-40ce-b688-471db30261be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:01.065179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.065150 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume" pod "43bd7df0-55e9-4aea-ad3e-324dd477c83c" (UID: "43bd7df0-55e9-4aea-ad3e-324dd477c83c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:01.065425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.065403 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bd7df0-55e9-4aea-ad3e-324dd477c83c-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:01.065510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.065431 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654bc5f1-11ca-40ce-b688-471db30261be-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:01.066743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.066716 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79" pod "654bc5f1-11ca-40ce-b688-471db30261be" (UID: "654bc5f1-11ca-40ce-b688-471db30261be"). InnerVolumeSpecName "kube-api-access-lps79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:01.067201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.067178 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw" pod "43bd7df0-55e9-4aea-ad3e-324dd477c83c" (UID: "43bd7df0-55e9-4aea-ad3e-324dd477c83c"). InnerVolumeSpecName "kube-api-access-dzgkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:01.166114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.166034 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerName="payload-pre-processing" containerID="cri-o://4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e" gracePeriod=25 Oct 30 15:24:01.166776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.166752 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzgkw\" (UniqueName: \"kubernetes.io/projected/43bd7df0-55e9-4aea-ad3e-324dd477c83c-kube-api-access-dzgkw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:01.166868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.166778 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lps79\" (UniqueName: \"kubernetes.io/projected/654bc5f1-11ca-40ce-b688-471db30261be-kube-api-access-lps79\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:01.664047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.664014 6554 generic.go:356] "Generic (PLEG): container finished" podID="654bc5f1-11ca-40ce-b688-471db30261be" containerID="e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f" exitCode=0 Oct 30 15:24:01.664196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.664059 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" Oct 30 15:24:01.664196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.664093 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" event={"ID":"654bc5f1-11ca-40ce-b688-471db30261be","Type":"ContainerDied","Data":"e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f"} Oct 30 15:24:01.664196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.664140 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l" event={"ID":"654bc5f1-11ca-40ce-b688-471db30261be","Type":"ContainerDied","Data":"00d6627df904da775d6fb8eb7767053de18042dbbf3cf288c2dc5f7dd4bc635a"} Oct 30 15:24:01.664196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.664162 6554 scope.go:122] "RemoveContainer" containerID="e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f" Oct 30 15:24:01.665904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.665879 6554 generic.go:356] "Generic (PLEG): container finished" podID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerID="60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289" exitCode=0 Oct 30 15:24:01.666026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.665928 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" Oct 30 15:24:01.666026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.665955 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" event={"ID":"43bd7df0-55e9-4aea-ad3e-324dd477c83c","Type":"ContainerDied","Data":"60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289"} Oct 30 15:24:01.666026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.666000 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46" event={"ID":"43bd7df0-55e9-4aea-ad3e-324dd477c83c","Type":"ContainerDied","Data":"959bd85eb0c2659aeab89dfedcc592d4c64a8eec235a6e645f5c7c57fdab3f95"} Oct 30 15:24:01.677647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.677628 6554 scope.go:122] "RemoveContainer" containerID="e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f" Oct 30 15:24:01.677974 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:01.677950 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f\": container with ID starting with e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f not found: ID does not exist" containerID="e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f" Oct 30 15:24:01.678145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.677980 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f"} err="failed to get container status \"e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f\": rpc error: code = NotFound desc = could not find container \"e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f\": container with ID starting with e1034b4f57d619b2d514278b5044db4edaafb8ef641a922d3161a5cbc811798f not found: ID does not exist" Oct 30 15:24:01.678145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.678000 6554 scope.go:122] "RemoveContainer" containerID="60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289" Oct 30 15:24:01.692122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.692103 6554 scope.go:122] "RemoveContainer" containerID="60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289" Oct 30 15:24:01.692391 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:01.692369 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289\": container with ID starting with 60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289 not found: ID does not exist" containerID="60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289" Oct 30 15:24:01.692470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.692396 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289"} err="failed to get container status \"60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289\": rpc error: code = NotFound desc = could not find container \"60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289\": container with ID starting with 60d32a305ce590efa784f98454e2200183b03158dd7f4f805267c47252981289 not found: ID does not exist" Oct 30 15:24:01.737482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.737458 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l"] Oct 30 15:24:01.750899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.750878 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-3bf5d2-54965ffbf9-bd95l"] Oct 30 15:24:01.777227 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.777201 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46"] Oct 30 15:24:01.781827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:01.781798 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-3bf5d2-d5b87788d-8vm46"] Oct 30 15:24:02.159456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:02.159422 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" path="/var/lib/kubelet/pods/43bd7df0-55e9-4aea-ad3e-324dd477c83c/volumes" Oct 30 15:24:02.160410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:02.160008 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654bc5f1-11ca-40ce-b688-471db30261be" path="/var/lib/kubelet/pods/654bc5f1-11ca-40ce-b688-471db30261be/volumes" Oct 30 15:24:02.449891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:02.449812 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv"] Oct 30 15:24:02.450072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:02.450051 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" containerID="cri-o://1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18" gracePeriod=30 Oct 30 15:24:04.225832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.225792 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb"] Oct 30 15:24:04.227003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.226981 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" Oct 30 15:24:04.227003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227004 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227021 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c7f03875-b078-4c81-b512-26e085ff6388" containerName="payload-pre-processing" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227029 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f03875-b078-4c81-b512-26e085ff6388" containerName="payload-pre-processing" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227046 6554 cpu_manager.go:414] "containerMap: removing container" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227054 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227079 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227087 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227103 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227110 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227126 6554 cpu_manager.go:414] "containerMap: removing container" podUID="654bc5f1-11ca-40ce-b688-471db30261be" containerName="payload-pre-processing" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227133 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654bc5f1-11ca-40ce-b688-471db30261be" containerName="payload-pre-processing" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227144 6554 cpu_manager.go:414] "containerMap: removing container" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerName="payload-processing" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227149 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerName="payload-processing" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227163 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227168 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" Oct 30 15:24:04.227194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227200 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" containerName="payload-processing" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227205 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" containerName="payload-processing" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227342 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227366 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f03875-b078-4c81-b512-26e085ff6388" containerName="payload-pre-processing" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227381 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e39c5bd-d7e5-4342-9663-12b1237020f2" containerName="istio-proxy" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227389 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b9e7c8-9e0d-4a10-a3c5-3906077e22aa" containerName="istio-proxy" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227395 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab402479-669a-4335-8c47-b7bce678d1d8" containerName="istio-proxy" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227400 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="654bc5f1-11ca-40ce-b688-471db30261be" containerName="payload-pre-processing" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227410 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="43bd7df0-55e9-4aea-ad3e-324dd477c83c" containerName="payload-processing" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227421 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c91760f-f50c-4ce7-800d-a8c40cee1ef0" containerName="istio-proxy" Oct 30 15:24:04.227761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.227432 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba806c4-36f9-4df7-8430-27caa0178955" containerName="payload-processing" Oct 30 15:24:04.232533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.232516 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.249421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.249398 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-100b2e-openshift-default-dockercfg-bsnrg\"" Oct 30 15:24:04.267459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.267433 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb"] Oct 30 15:24:04.316570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.316528 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.316751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.316574 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.316751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.316679 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.316844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.316793 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.316844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.316832 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f682c\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.316925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.316905 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.317019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.317007 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.317055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.317028 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.317105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.317058 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.418666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418636 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.418862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418684 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f682c\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.418862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418735 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.418862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418818 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.418862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418835 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.418862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418853 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418924 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418953 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.418990 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.419302 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.419307 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.419448 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.419483 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.419636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.419614 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.421154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.421133 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.421272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.421251 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.444073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.444050 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f682c\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.444173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.444120 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token\") pod \"e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.546526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.546470 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:04.699953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.699919 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:04.875920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.875890 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb"] Oct 30 15:24:04.876044 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:24:04.876030 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab2baa3_9253_4029_bbdf_6ef9cbd5faf5.slice/crio-02106d6b24df3777b20d9fd9416a374559a2cb95537ac429f03191a0514a938f WatchSource:0}: Error finding container 02106d6b24df3777b20d9fd9416a374559a2cb95537ac429f03191a0514a938f: Status 404 returned error can't find the container with id 02106d6b24df3777b20d9fd9416a374559a2cb95537ac429f03191a0514a938f Oct 30 15:24:04.878711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.878679 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:24:04.878800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.878738 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:24:04.878800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:04.878770 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:24:05.705710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:05.705672 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" event={"ID":"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5","Type":"ContainerStarted","Data":"4d7b3fdd1b0f4b2adafe505c4a5303cd024bac869e1dd1292646c741f1ffd86c"} Oct 30 15:24:05.706150 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:05.705732 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" event={"ID":"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5","Type":"ContainerStarted","Data":"02106d6b24df3777b20d9fd9416a374559a2cb95537ac429f03191a0514a938f"} Oct 30 15:24:05.784138 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:05.784074 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podStartSLOduration=1.78405632 podStartE2EDuration="1.78405632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:24:05.780012281 +0000 UTC m=+2149.739053688" watchObservedRunningTime="2026-10-30 15:24:05.78405632 +0000 UTC m=+2149.743097732" Oct 30 15:24:06.546666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:06.546633 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:06.547947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:06.547925 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 15:24:06.548035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:06.547979 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 15:24:07.547472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.547442 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 15:24:07.547872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.547492 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 15:24:07.718237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.718204 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:24:07.722181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.722150 6554 generic.go:356] "Generic (PLEG): container finished" podID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerID="1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18" exitCode=0 Oct 30 15:24:07.722303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.722198 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" Oct 30 15:24:07.722303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.722228 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" event={"ID":"46ed6f78-cca3-4e47-b41b-9f0e2200ce88","Type":"ContainerDied","Data":"1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18"} Oct 30 15:24:07.722406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.722306 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv" event={"ID":"46ed6f78-cca3-4e47-b41b-9f0e2200ce88","Type":"ContainerDied","Data":"d08dd0ea3db2d7f0a5a2c30643914a88a6d1c248952b305d90145a1e40271622"} Oct 30 15:24:07.722406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.722331 6554 scope.go:122] "RemoveContainer" containerID="1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18" Oct 30 15:24:07.733047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.733029 6554 scope.go:122] "RemoveContainer" containerID="28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747" Oct 30 15:24:07.742457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.742439 6554 scope.go:122] "RemoveContainer" containerID="1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18" Oct 30 15:24:07.742744 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:07.742720 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18\": container with ID starting with 1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18 not found: ID does not exist" containerID="1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18" Oct 30 15:24:07.742803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.742758 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18"} err="failed to get container status \"1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18\": rpc error: code = NotFound desc = could not find container \"1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18\": container with ID starting with 1fafea62daa4b0728a6a3e96ac8113cfca6c0a7e1e933c068dd47e90d974ab18 not found: ID does not exist" Oct 30 15:24:07.742803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.742782 6554 scope.go:122] "RemoveContainer" containerID="28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747" Oct 30 15:24:07.743055 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:07.743034 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747\": container with ID starting with 28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747 not found: ID does not exist" containerID="28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747" Oct 30 15:24:07.743111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.743061 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747"} err="failed to get container status \"28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747\": rpc error: code = NotFound desc = could not find container \"28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747\": container with ID starting with 28e45c40b97b7646555ea7bfecfa34e0f40053df355f6d131515a9be2869a747 not found: ID does not exist" Oct 30 15:24:07.905092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905061 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905141 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905160 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905178 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905206 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905254 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905324 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905404 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:07.905601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905439 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:07.905601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905445 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905517 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data\") pod \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\" (UID: \"46ed6f78-cca3-4e47-b41b-9f0e2200ce88\") " Oct 30 15:24:07.905885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905714 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:07.905998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.905975 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:07.906069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.906018 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:07.906539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.906517 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.906539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.906539 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.906690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.906551 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.906690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.906564 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.906690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.906573 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:07.907834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.907805 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:07.908003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.907929 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:24:07.908319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.908299 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:07.909135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:07.909111 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw" pod "46ed6f78-cca3-4e47-b41b-9f0e2200ce88" (UID: "46ed6f78-cca3-4e47-b41b-9f0e2200ce88"). InnerVolumeSpecName "kube-api-access-hshbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:08.018431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.018180 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hshbw\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-kube-api-access-hshbw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:08.018431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.018237 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:08.018431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.018253 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:08.018431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.018275 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46ed6f78-cca3-4e47-b41b-9f0e2200ce88-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:08.080799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.080705 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv"] Oct 30 15:24:08.084219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.084196 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3bf5d2-openshift-default-5c676b6f6c-zlrjv"] Oct 30 15:24:08.180292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.180255 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" path="/var/lib/kubelet/pods/46ed6f78-cca3-4e47-b41b-9f0e2200ce88/volumes" Oct 30 15:24:08.546959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.546929 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 15:24:08.547105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:08.546980 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 15:24:09.206620 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:09.206578 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:24:09.547034 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:09.546956 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 15:24:09.547034 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:09.547005 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 15:24:10.547224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:10.547189 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body= Oct 30 15:24:10.547693 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:10.547247 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" Oct 30 15:24:11.567394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:11.567344 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:11.568117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:11.568091 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:11.569063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:11.569040 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:24:14.278901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:14.278870 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8"] Oct 30 15:24:14.700224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:14.700183 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:24:19.279984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.279944 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" podUID="73c9b2a7-b631-48ad-ab9c-48e9934ae680" containerName="payload-processing" containerID="cri-o://23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f" gracePeriod=25 Oct 30 15:24:19.564042 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.564015 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:24:19.626294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.626255 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr\" (UniqueName: \"kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr\") pod \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " Oct 30 15:24:19.628451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.628422 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr" pod "73c9b2a7-b631-48ad-ab9c-48e9934ae680" (UID: "73c9b2a7-b631-48ad-ab9c-48e9934ae680"). InnerVolumeSpecName "kube-api-access-fxstr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:19.727425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.727401 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume\") pod \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\" (UID: \"73c9b2a7-b631-48ad-ab9c-48e9934ae680\") " Oct 30 15:24:19.727722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.727701 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume" pod "73c9b2a7-b631-48ad-ab9c-48e9934ae680" (UID: "73c9b2a7-b631-48ad-ab9c-48e9934ae680"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:19.727835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.727821 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxstr\" (UniqueName: \"kubernetes.io/projected/73c9b2a7-b631-48ad-ab9c-48e9934ae680-kube-api-access-fxstr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:19.727891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.727838 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c9b2a7-b631-48ad-ab9c-48e9934ae680-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:19.816018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.815962 6554 generic.go:356] "Generic (PLEG): container finished" podID="73c9b2a7-b631-48ad-ab9c-48e9934ae680" containerID="23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f" exitCode=0 Oct 30 15:24:19.816018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.816006 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" Oct 30 15:24:19.816173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.816041 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" event={"ID":"73c9b2a7-b631-48ad-ab9c-48e9934ae680","Type":"ContainerDied","Data":"23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f"} Oct 30 15:24:19.816173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.816094 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8" event={"ID":"73c9b2a7-b631-48ad-ab9c-48e9934ae680","Type":"ContainerDied","Data":"71eb94d5dd0ba662ebbe9f9dbed84056792471e20db2322ccb566bf0d69fe1c4"} Oct 30 15:24:19.816173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.816117 6554 scope.go:122] "RemoveContainer" containerID="23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f" Oct 30 15:24:19.826561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.826544 6554 scope.go:122] "RemoveContainer" containerID="23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f" Oct 30 15:24:19.826807 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:19.826784 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f\": container with ID starting with 23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f not found: ID does not exist" containerID="23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f" Oct 30 15:24:19.826880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.826814 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f"} err="failed to get container status \"23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f\": rpc error: code = NotFound desc = could not find container \"23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f\": container with ID starting with 23d5c3cb2b196036793f3597ae0e12b8364a1bba7e18070515e2e05133c2583f not found: ID does not exist" Oct 30 15:24:19.872834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.872803 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8"] Oct 30 15:24:19.877185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:19.877164 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0a4f54-6477968bb-6bfk8"] Oct 30 15:24:20.155851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:20.155824 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c9b2a7-b631-48ad-ab9c-48e9934ae680" path="/var/lib/kubelet/pods/73c9b2a7-b631-48ad-ab9c-48e9934ae680/volumes" Oct 30 15:24:21.437776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.437715 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:24:21.456806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.456776 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume\") pod \"94dc6f7c-0a67-48af-9563-1569e8217e12\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " Oct 30 15:24:21.456960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.456944 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm\" (UniqueName: \"kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm\") pod \"94dc6f7c-0a67-48af-9563-1569e8217e12\" (UID: \"94dc6f7c-0a67-48af-9563-1569e8217e12\") " Oct 30 15:24:21.457109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.457088 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume" pod "94dc6f7c-0a67-48af-9563-1569e8217e12" (UID: "94dc6f7c-0a67-48af-9563-1569e8217e12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:21.457481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.457462 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94dc6f7c-0a67-48af-9563-1569e8217e12-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:21.458982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.458963 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm" pod "94dc6f7c-0a67-48af-9563-1569e8217e12" (UID: "94dc6f7c-0a67-48af-9563-1569e8217e12"). InnerVolumeSpecName "kube-api-access-7mfmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:21.559298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.559263 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mfmm\" (UniqueName: \"kubernetes.io/projected/94dc6f7c-0a67-48af-9563-1569e8217e12-kube-api-access-7mfmm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:21.838963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.838892 6554 generic.go:356] "Generic (PLEG): container finished" podID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerID="4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e" exitCode=0 Oct 30 15:24:21.838963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.838935 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" Oct 30 15:24:21.839131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.838968 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" event={"ID":"94dc6f7c-0a67-48af-9563-1569e8217e12","Type":"ContainerDied","Data":"4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e"} Oct 30 15:24:21.839131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.839038 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d" event={"ID":"94dc6f7c-0a67-48af-9563-1569e8217e12","Type":"ContainerDied","Data":"7552e57f163ccc381c452c6f45002ce1d4257cfa1199e2b999b03470e8e843fe"} Oct 30 15:24:21.839131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.839061 6554 scope.go:122] "RemoveContainer" containerID="4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e" Oct 30 15:24:21.850207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.850191 6554 scope.go:122] "RemoveContainer" containerID="4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e" Oct 30 15:24:21.850478 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:21.850459 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e\": container with ID starting with 4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e not found: ID does not exist" containerID="4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e" Oct 30 15:24:21.850541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.850486 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e"} err="failed to get container status \"4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e\": rpc error: code = NotFound desc = could not find container \"4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e\": container with ID starting with 4d05dc1a8deaf812db236bbea1c0333ca103c6d7ae240c1964238a4ee6c5618e not found: ID does not exist" Oct 30 15:24:21.896596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.896575 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d"] Oct 30 15:24:21.909249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:21.909229 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d84bc6-5f599dc57f-9wk4d"] Oct 30 15:24:22.155203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:22.155175 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" path="/var/lib/kubelet/pods/94dc6f7c-0a67-48af-9563-1569e8217e12/volumes" Oct 30 15:24:37.445043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:37.444852 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77"] Oct 30 15:24:37.446482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:37.446421 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" containerID="cri-o://4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb" gracePeriod=30 Oct 30 15:24:39.203711 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:39.203674 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:24:40.567462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:40.567428 6554 patch_prober.go:29] interesting pod/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:24:40.567863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:40.567491 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:24:42.724696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.724630 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:24:42.780830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.780807 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.780991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.780869 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.780991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.780896 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.780991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.780927 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.781169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781002 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.781169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781072 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.781169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781131 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.781333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781184 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.781333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781199 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:42.781333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781279 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs\") pod \"6a81a0ea-b425-4207-9003-e3d8c748260d\" (UID: \"6a81a0ea-b425-4207-9003-e3d8c748260d\") " Oct 30 15:24:42.781333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781292 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:42.781668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781461 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:42.781668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781574 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:42.781668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.781607 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:42.782036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.782013 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6a81a0ea-b425-4207-9003-e3d8c748260d-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.782118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.782042 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.782118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.782058 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.782118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.782073 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.782118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.782088 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.783282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.783260 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:24:42.783458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.783415 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:24:42.783584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.783561 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:42.783638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.783598 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82" pod "6a81a0ea-b425-4207-9003-e3d8c748260d" (UID: "6a81a0ea-b425-4207-9003-e3d8c748260d"). InnerVolumeSpecName "kube-api-access-njl82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:42.883716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.883684 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njl82\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-kube-api-access-njl82\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.883716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.883710 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.883716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.883721 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:42.883928 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:42.883729 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6a81a0ea-b425-4207-9003-e3d8c748260d-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:43.004000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.003924 6554 generic.go:356] "Generic (PLEG): container finished" podID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerID="4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb" exitCode=0 Oct 30 15:24:43.004000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.003976 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" Oct 30 15:24:43.004000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.003984 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" event={"ID":"6a81a0ea-b425-4207-9003-e3d8c748260d","Type":"ContainerDied","Data":"4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb"} Oct 30 15:24:43.004207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.004024 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77" event={"ID":"6a81a0ea-b425-4207-9003-e3d8c748260d","Type":"ContainerDied","Data":"8e408c7f66bc70ae4d54d29b5628160b53dcd1af14f6ea2b1f36a1c624c1639a"} Oct 30 15:24:43.004207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.004047 6554 scope.go:122] "RemoveContainer" containerID="4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb" Oct 30 15:24:43.016498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.016432 6554 scope.go:122] "RemoveContainer" containerID="d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa" Oct 30 15:24:43.026298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.026281 6554 scope.go:122] "RemoveContainer" containerID="4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb" Oct 30 15:24:43.026576 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:43.026553 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb\": container with ID starting with 4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb not found: ID does not exist" containerID="4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb" Oct 30 15:24:43.026674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.026584 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb"} err="failed to get container status \"4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb\": rpc error: code = NotFound desc = could not find container \"4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb\": container with ID starting with 4a897248e339fc70ba3303c22a76281d05e06861a16ccc57352af0d5941d00cb not found: ID does not exist" Oct 30 15:24:43.026674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.026605 6554 scope.go:122] "RemoveContainer" containerID="d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa" Oct 30 15:24:43.026826 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:43.026808 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa\": container with ID starting with d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa not found: ID does not exist" containerID="d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa" Oct 30 15:24:43.026870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.026833 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa"} err="failed to get container status \"d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa\": rpc error: code = NotFound desc = could not find container \"d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa\": container with ID starting with d4aac54072795a56b25fbc0cb7925e0f140fd5852ba7d002289afbd5f4d29eaa not found: ID does not exist" Oct 30 15:24:43.062227 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.062199 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77"] Oct 30 15:24:43.065857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:43.065835 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d84bc6-openshift-default-7d8499cdc5-vqp77"] Oct 30 15:24:44.157773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:44.157732 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" path="/var/lib/kubelet/pods/6a81a0ea-b425-4207-9003-e3d8c748260d/volumes" Oct 30 15:24:50.696704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:50.696664 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg"] Oct 30 15:24:50.697226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:50.696885 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" podUID="006ec327-c900-4c9b-92ab-185d124e6377" containerName="maas-api" containerID="cri-o://d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681" gracePeriod=30 Oct 30 15:24:51.027802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.027771 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:24:51.048671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.048644 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls\") pod \"006ec327-c900-4c9b-92ab-185d124e6377\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " Oct 30 15:24:51.048671 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.048674 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs\") pod \"006ec327-c900-4c9b-92ab-185d124e6377\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " Oct 30 15:24:51.048837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.048762 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca\") pod \"006ec327-c900-4c9b-92ab-185d124e6377\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " Oct 30 15:24:51.048979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.048959 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r\" (UniqueName: \"kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r\") pod \"006ec327-c900-4c9b-92ab-185d124e6377\" (UID: \"006ec327-c900-4c9b-92ab-185d124e6377\") " Oct 30 15:24:51.049184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.049156 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca" pod "006ec327-c900-4c9b-92ab-185d124e6377" (UID: "006ec327-c900-4c9b-92ab-185d124e6377"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:24:51.049680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.049655 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/006ec327-c900-4c9b-92ab-185d124e6377-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:51.050861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.050835 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs" pod "006ec327-c900-4c9b-92ab-185d124e6377" (UID: "006ec327-c900-4c9b-92ab-185d124e6377"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:24:51.051032 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.051011 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls" pod "006ec327-c900-4c9b-92ab-185d124e6377" (UID: "006ec327-c900-4c9b-92ab-185d124e6377"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:24:51.051119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.051099 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r" pod "006ec327-c900-4c9b-92ab-185d124e6377" (UID: "006ec327-c900-4c9b-92ab-185d124e6377"). InnerVolumeSpecName "kube-api-access-d4g5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:24:51.064168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.064024 6554 generic.go:356] "Generic (PLEG): container finished" podID="006ec327-c900-4c9b-92ab-185d124e6377" containerID="d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681" exitCode=0 Oct 30 15:24:51.064168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.064070 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" Oct 30 15:24:51.064168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.064110 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" event={"ID":"006ec327-c900-4c9b-92ab-185d124e6377","Type":"ContainerDied","Data":"d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681"} Oct 30 15:24:51.064288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.064174 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg" event={"ID":"006ec327-c900-4c9b-92ab-185d124e6377","Type":"ContainerDied","Data":"144ce494fdad1b4c0170f6d17bae9f424939b404988f051ac317e1c9f3c772ae"} Oct 30 15:24:51.064288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.064207 6554 scope.go:122] "RemoveContainer" containerID="d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681" Oct 30 15:24:51.078999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.078976 6554 scope.go:122] "RemoveContainer" containerID="d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681" Oct 30 15:24:51.079297 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:24:51.079271 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681\": container with ID starting with d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681 not found: ID does not exist" containerID="d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681" Oct 30 15:24:51.079438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.079302 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681"} err="failed to get container status \"d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681\": rpc error: code = NotFound desc = could not find container \"d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681\": container with ID starting with d753ab593cf272bacf862056f8f0adf2afa66011289981e206d19d67f559c681 not found: ID does not exist" Oct 30 15:24:51.138470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.138407 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg"] Oct 30 15:24:51.144296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.144267 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0a4f54-7759485978-727pg"] Oct 30 15:24:51.152643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.151694 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4g5r\" (UniqueName: \"kubernetes.io/projected/006ec327-c900-4c9b-92ab-185d124e6377-kube-api-access-d4g5r\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:51.152643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.151728 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:51.152643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:51.151742 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/006ec327-c900-4c9b-92ab-185d124e6377-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:24:52.159887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:52.159853 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006ec327-c900-4c9b-92ab-185d124e6377" path="/var/lib/kubelet/pods/006ec327-c900-4c9b-92ab-185d124e6377/volumes" Oct 30 15:24:59.020623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:59.020575 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb"] Oct 30 15:24:59.021010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:24:59.020840 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" containerID="cri-o://da2954ecbe0666b67560ddd3da459c86aced9a0febed9e7f5e51abbb936060bc" gracePeriod=30 Oct 30 15:25:04.226162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.225901 6554 generic.go:356] "Generic (PLEG): container finished" podID="be847216-389a-4225-9f40-42e6a1a8adce" containerID="da2954ecbe0666b67560ddd3da459c86aced9a0febed9e7f5e51abbb936060bc" exitCode=0 Oct 30 15:25:04.226621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.226143 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" event={"ID":"be847216-389a-4225-9f40-42e6a1a8adce","Type":"ContainerDied","Data":"da2954ecbe0666b67560ddd3da459c86aced9a0febed9e7f5e51abbb936060bc"} Oct 30 15:25:04.226621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.226295 6554 scope.go:122] "RemoveContainer" containerID="47adce0412e53237094a9f17560fe9172e51096bd39249480c50f9b6bc075a52" Oct 30 15:25:04.329281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.329256 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:25:04.412753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.412721 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413153 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413251 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413283 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413445 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413487 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413560 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413559 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.413723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413636 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413666 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo\") pod \"be847216-389a-4225-9f40-42e6a1a8adce\" (UID: \"be847216-389a-4225-9f40-42e6a1a8adce\") " Oct 30 15:25:04.413835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.413785 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.415385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.414122 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:04.415385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.414418 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.415385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.414441 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be847216-389a-4225-9f40-42e6a1a8adce-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.415385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.414456 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.415385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.414709 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.415385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.415158 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.416184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.416148 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:04.417501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.417478 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:25:04.418720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.418700 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "kube-api-access-82hxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:04.419170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.419153 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy" pod "be847216-389a-4225-9f40-42e6a1a8adce" (UID: "be847216-389a-4225-9f40-42e6a1a8adce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:25:04.516043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.516007 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.516043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.516044 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82hxk\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-kube-api-access-82hxk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.516267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.516060 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.516267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.516075 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be847216-389a-4225-9f40-42e6a1a8adce-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.516267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.516087 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be847216-389a-4225-9f40-42e6a1a8adce-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:04.516267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:04.516100 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be847216-389a-4225-9f40-42e6a1a8adce-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:05.234230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:05.234201 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" Oct 30 15:25:05.234769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:05.234245 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb" event={"ID":"be847216-389a-4225-9f40-42e6a1a8adce","Type":"ContainerDied","Data":"a9f0af090b97db989f832c7e75f41e5f7f507d178329eed11daddf84d669d9e6"} Oct 30 15:25:05.234769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:05.234298 6554 scope.go:122] "RemoveContainer" containerID="da2954ecbe0666b67560ddd3da459c86aced9a0febed9e7f5e51abbb936060bc" Oct 30 15:25:05.305097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:05.305070 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb"] Oct 30 15:25:05.318674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:05.318650 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0a4f54-openshift-default-79d98b5cc-bbshb"] Oct 30 15:25:06.155569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.155540 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be847216-389a-4225-9f40-42e6a1a8adce" path="/var/lib/kubelet/pods/be847216-389a-4225-9f40-42e6a1a8adce/volumes" Oct 30 15:25:06.225336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.225301 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8"] Oct 30 15:25:06.226383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226320 6554 cpu_manager.go:414] "containerMap: removing container" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226384 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226415 6554 cpu_manager.go:414] "containerMap: removing container" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226424 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226452 6554 cpu_manager.go:414] "containerMap: removing container" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226460 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226491 6554 cpu_manager.go:414] "containerMap: removing container" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerName="payload-pre-processing" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226498 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerName="payload-pre-processing" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226509 6554 cpu_manager.go:414] "containerMap: removing container" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" Oct 30 15:25:06.226512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226516 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226540 6554 cpu_manager.go:414] "containerMap: removing container" podUID="73c9b2a7-b631-48ad-ab9c-48e9934ae680" containerName="payload-processing" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226547 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c9b2a7-b631-48ad-ab9c-48e9934ae680" containerName="payload-processing" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226558 6554 cpu_manager.go:414] "containerMap: removing container" podUID="006ec327-c900-4c9b-92ab-185d124e6377" containerName="maas-api" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226565 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006ec327-c900-4c9b-92ab-185d124e6377" containerName="maas-api" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226583 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226591 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226603 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226608 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226746 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226761 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c9b2a7-b631-48ad-ab9c-48e9934ae680" containerName="payload-processing" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226770 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="006ec327-c900-4c9b-92ab-185d124e6377" containerName="maas-api" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226777 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226784 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a81a0ea-b425-4207-9003-e3d8c748260d" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226793 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226800 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ed6f78-cca3-4e47-b41b-9f0e2200ce88" containerName="istio-proxy" Oct 30 15:25:06.226869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.226810 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="94dc6f7c-0a67-48af-9563-1569e8217e12" containerName="payload-pre-processing" Oct 30 15:25:06.231965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.231941 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.249260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.249236 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8"] Oct 30 15:25:06.250364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.250321 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-3f20f9\"" Oct 30 15:25:06.259216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.259189 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-3f20f9-dockercfg-9r7h4\"" Oct 30 15:25:06.290935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.290907 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm"] Oct 30 15:25:06.292121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.292096 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="be847216-389a-4225-9f40-42e6a1a8adce" containerName="istio-proxy" Oct 30 15:25:06.306795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.306763 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.330374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.314572 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm"] Oct 30 15:25:06.356587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.346302 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rxqj\" (UniqueName: \"kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj\") pod \"payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.356587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.346481 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.472795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.472753 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.472982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.472885 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpjts\" (UniqueName: \"kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts\") pod \"payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.473082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.473066 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.473173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.473158 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rxqj\" (UniqueName: \"kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj\") pod \"payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.473515 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.473487 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.497977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.497953 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rxqj\" (UniqueName: \"kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj\") pod \"payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.548209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.548184 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:06.574896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.574868 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpjts\" (UniqueName: \"kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts\") pod \"payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.575032 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.574961 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.575627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.575608 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.599306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.599279 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpjts\" (UniqueName: \"kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts\") pod \"payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.716568 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.716494 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:06.857155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:06.857129 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8"] Oct 30 15:25:06.858113 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:25:06.858087 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d8c605_42eb_4c52_850c_c7e6863c6753.slice/crio-84eafa53990003e36a6427b8afc3c20389ad4d6de5ec035b5969e5e26dabb216 WatchSource:0}: Error finding container 84eafa53990003e36a6427b8afc3c20389ad4d6de5ec035b5969e5e26dabb216: Status 404 returned error can't find the container with id 84eafa53990003e36a6427b8afc3c20389ad4d6de5ec035b5969e5e26dabb216 Oct 30 15:25:07.053025 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:25:07.053000 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d8f8af_2bbc_47b5_ae69_a05f33866190.slice/crio-0cc2e7ab71db95a70657191a010b0c22d29913e4b5ac26a80ffed70087a21feb WatchSource:0}: Error finding container 0cc2e7ab71db95a70657191a010b0c22d29913e4b5ac26a80ffed70087a21feb: Status 404 returned error can't find the container with id 0cc2e7ab71db95a70657191a010b0c22d29913e4b5ac26a80ffed70087a21feb Oct 30 15:25:07.053603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.053580 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm"] Oct 30 15:25:07.254194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.254159 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" event={"ID":"45d8c605-42eb-4c52-850c-c7e6863c6753","Type":"ContainerStarted","Data":"a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730"} Oct 30 15:25:07.254603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.254214 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" event={"ID":"45d8c605-42eb-4c52-850c-c7e6863c6753","Type":"ContainerStarted","Data":"84eafa53990003e36a6427b8afc3c20389ad4d6de5ec035b5969e5e26dabb216"} Oct 30 15:25:07.254603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.254246 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:07.256021 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.255998 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" event={"ID":"b3d8f8af-2bbc-47b5-ae69-a05f33866190","Type":"ContainerStarted","Data":"96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b"} Oct 30 15:25:07.256137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.256037 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" event={"ID":"b3d8f8af-2bbc-47b5-ae69-a05f33866190","Type":"ContainerStarted","Data":"0cc2e7ab71db95a70657191a010b0c22d29913e4b5ac26a80ffed70087a21feb"} Oct 30 15:25:07.256200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.256141 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:07.300926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.300842 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" podStartSLOduration=1.300830676 podStartE2EDuration="1.300830676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:07.299978544 +0000 UTC m=+2211.259019950" watchObservedRunningTime="2026-10-30 15:25:07.300830676 +0000 UTC m=+2211.259872081" Oct 30 15:25:07.346030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:07.345982 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" podStartSLOduration=1.345969845 podStartE2EDuration="1.345969845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:07.342712741 +0000 UTC m=+2211.301754143" watchObservedRunningTime="2026-10-30 15:25:07.345969845 +0000 UTC m=+2211.305011277" Oct 30 15:25:09.202316 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:25:09.202271 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:25:18.265860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:18.265815 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:25:18.266450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:18.265902 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:25:32.850408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.850378 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl"] Oct 30 15:25:32.856556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.856536 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.876851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.876827 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-985bf6-openshift-default-dockercfg-wt6nx\"" Oct 30 15:25:32.892079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.892049 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl"] Oct 30 15:25:32.996537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996513 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqb9\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996559 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996676 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996704 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996730 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996816 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996777 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996879 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996897 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:32.996962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:32.996923 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098287 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqb9\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098332 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098423 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098451 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098578 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098632 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098734 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098766 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098793 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098818 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.098921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.098834 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.099090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.099063 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.099169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.099086 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.099603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.099581 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.100838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.100798 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.101226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.101206 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.123435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.123411 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.123644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.123621 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqb9\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.170306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.170285 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:33.493242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.493217 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl"] Oct 30 15:25:33.494477 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:25:33.494448 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630789f8_ab60_4ae5_b3b6_5bb64fb62223.slice/crio-ec0910f55666494477396343e14ac663ff24f2ff3c209c2a9e8beb22f448f655 WatchSource:0}: Error finding container ec0910f55666494477396343e14ac663ff24f2ff3c209c2a9e8beb22f448f655: Status 404 returned error can't find the container with id ec0910f55666494477396343e14ac663ff24f2ff3c209c2a9e8beb22f448f655 Oct 30 15:25:33.497055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.497022 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:25:33.497129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.497091 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:25:33.497129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:33.497120 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:25:34.492387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:34.488985 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" event={"ID":"630789f8-ab60-4ae5-b3b6-5bb64fb62223","Type":"ContainerStarted","Data":"45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c"} Oct 30 15:25:34.492387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:34.489064 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" event={"ID":"630789f8-ab60-4ae5-b3b6-5bb64fb62223","Type":"ContainerStarted","Data":"ec0910f55666494477396343e14ac663ff24f2ff3c209c2a9e8beb22f448f655"} Oct 30 15:25:34.540932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:34.540875 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podStartSLOduration=2.540861247 podStartE2EDuration="2.540861247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:25:34.537102498 +0000 UTC m=+2238.496143906" watchObservedRunningTime="2026-10-30 15:25:34.540861247 +0000 UTC m=+2238.499902652" Oct 30 15:25:35.176864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:35.176831 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:35.180434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:35.180402 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 15:25:35.180559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:35.180463 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 15:25:36.176274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:36.174683 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 15:25:36.176274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:36.174739 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 15:25:37.174374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:37.173393 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 15:25:37.174374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:37.173457 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 15:25:38.178430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:38.174727 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 15:25:38.178430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:38.174782 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 15:25:39.178941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:39.178901 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body= Oct 30 15:25:39.179460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:39.178958 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" Oct 30 15:25:39.207403 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:25:39.207345 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:25:40.175585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:40.175553 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:40.175941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:40.175918 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:40.176495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:40.176480 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:25:49.978930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:49.978888 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d"] Oct 30 15:25:50.012511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:50.012476 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq"] Oct 30 15:25:54.980277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:54.980234 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" podUID="5b8a4ed0-e17a-437a-9eba-91356b584128" containerName="payload-processing" containerID="cri-o://7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d" gracePeriod=25 Oct 30 15:25:55.012799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.012763 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerName="payload-pre-processing" containerID="cri-o://02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c" gracePeriod=25 Oct 30 15:25:55.255537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.255498 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:25:55.432181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.432151 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6\" (UniqueName: \"kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6\") pod \"5b8a4ed0-e17a-437a-9eba-91356b584128\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " Oct 30 15:25:55.432343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.432241 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume\") pod \"5b8a4ed0-e17a-437a-9eba-91356b584128\" (UID: \"5b8a4ed0-e17a-437a-9eba-91356b584128\") " Oct 30 15:25:55.432632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.432609 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume" pod "5b8a4ed0-e17a-437a-9eba-91356b584128" (UID: "5b8a4ed0-e17a-437a-9eba-91356b584128"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:25:55.432782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.432767 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b8a4ed0-e17a-437a-9eba-91356b584128-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:55.434460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.434437 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6" pod "5b8a4ed0-e17a-437a-9eba-91356b584128" (UID: "5b8a4ed0-e17a-437a-9eba-91356b584128"). InnerVolumeSpecName "kube-api-access-wz7t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:25:55.533820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.533765 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7t6\" (UniqueName: \"kubernetes.io/projected/5b8a4ed0-e17a-437a-9eba-91356b584128-kube-api-access-wz7t6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:25:55.662488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.662459 6554 generic.go:356] "Generic (PLEG): container finished" podID="5b8a4ed0-e17a-437a-9eba-91356b584128" containerID="7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d" exitCode=0 Oct 30 15:25:55.662586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.662506 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" Oct 30 15:25:55.662586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.662540 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" event={"ID":"5b8a4ed0-e17a-437a-9eba-91356b584128","Type":"ContainerDied","Data":"7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d"} Oct 30 15:25:55.662661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.662591 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d" event={"ID":"5b8a4ed0-e17a-437a-9eba-91356b584128","Type":"ContainerDied","Data":"1348e35f8e20172c2a0839c9096a8ac6b4bafbf824842b418a9b6988516e0769"} Oct 30 15:25:55.662661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.662612 6554 scope.go:122] "RemoveContainer" containerID="7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d" Oct 30 15:25:55.673508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.673493 6554 scope.go:122] "RemoveContainer" containerID="7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d" Oct 30 15:25:55.673761 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:25:55.673743 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d\": container with ID starting with 7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d not found: ID does not exist" containerID="7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d" Oct 30 15:25:55.673840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.673765 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d"} err="failed to get container status \"7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d\": rpc error: code = NotFound desc = could not find container \"7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d\": container with ID starting with 7593d63f7e98736359efc8ce9543aa1beb31b2dc77d7ec09c561dbd6d654096d not found: ID does not exist" Oct 30 15:25:55.719493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.719473 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d"] Oct 30 15:25:55.733001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:55.732979 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a9275a-7c779996bb-b8k5d"] Oct 30 15:25:56.155737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:56.155703 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b8a4ed0-e17a-437a-9eba-91356b584128" path="/var/lib/kubelet/pods/5b8a4ed0-e17a-437a-9eba-91356b584128/volumes" Oct 30 15:25:57.197186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:25:57.197104 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:07.196972 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:07.196936 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:09.203514 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:09.203474 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:26:09.455328 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:09.455247 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p"] Oct 30 15:26:09.455554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:09.455478 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" containerID="cri-o://cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0" gracePeriod=30 Oct 30 15:26:14.751968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.751941 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:26:14.843750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.843716 6554 generic.go:356] "Generic (PLEG): container finished" podID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerID="cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0" exitCode=0 Oct 30 15:26:14.843942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.843775 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" Oct 30 15:26:14.843942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.843778 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" event={"ID":"55618dab-3ed3-40bc-8f0e-68e4bc6893a0","Type":"ContainerDied","Data":"cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0"} Oct 30 15:26:14.843942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.843896 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p" event={"ID":"55618dab-3ed3-40bc-8f0e-68e4bc6893a0","Type":"ContainerDied","Data":"c537a9b37b52d4de942602c95e0ce071f5bd21081a7a4f814272bd10e2819ec4"} Oct 30 15:26:14.843942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.843917 6554 scope.go:122] "RemoveContainer" containerID="cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0" Oct 30 15:26:14.858263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.858241 6554 scope.go:122] "RemoveContainer" containerID="cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0" Oct 30 15:26:14.858608 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:14.858577 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0\": container with ID starting with cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0 not found: ID does not exist" containerID="cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0" Oct 30 15:26:14.858769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.858617 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0"} err="failed to get container status \"cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0\": rpc error: code = NotFound desc = could not find container \"cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0\": container with ID starting with cb5b2c65ee46e47a5f91767bb86ef88803b4898a04ceaa125a3c63fe9530a2e0 not found: ID does not exist" Oct 30 15:26:14.917896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.917873 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.917927 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.917966 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918028 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918086 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918114 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918185 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.918276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918242 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918287 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918343 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket\") pod \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\" (UID: \"55618dab-3ed3-40bc-8f0e-68e4bc6893a0\") " Oct 30 15:26:14.918616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918546 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:14.918890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918802 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.919014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.918982 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.919156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.919136 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.919265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.919160 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.919265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.919176 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.919265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.919189 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:14.919265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.919245 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:14.920500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.920470 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:26:14.920622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.920519 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.920718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.920701 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "kube-api-access-k4jzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:14.920957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:14.920938 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy" pod "55618dab-3ed3-40bc-8f0e-68e4bc6893a0" (UID: "55618dab-3ed3-40bc-8f0e-68e4bc6893a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:26:15.020305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.020281 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.020305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.020306 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.020477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.020315 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.020477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.020325 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jzp\" (UniqueName: \"kubernetes.io/projected/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-kube-api-access-k4jzp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.020477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.020337 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55618dab-3ed3-40bc-8f0e-68e4bc6893a0-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.198795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.198762 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p"] Oct 30 15:26:15.205030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.205004 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a9275a-openshift-default-7df797b445-khm7p"] Oct 30 15:26:15.277507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.277441 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:26:15.431193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.431158 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume\") pod \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " Oct 30 15:26:15.431442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.431422 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2\" (UniqueName: \"kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2\") pod \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\" (UID: \"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e\") " Oct 30 15:26:15.432925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.432896 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume" pod "5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" (UID: "5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:15.434843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.434819 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2" pod "5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" (UID: "5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e"). InnerVolumeSpecName "kube-api-access-hbtk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:15.533480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.533412 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.533480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.533437 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbtk2\" (UniqueName: \"kubernetes.io/projected/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e-kube-api-access-hbtk2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:15.851452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.851340 6554 generic.go:356] "Generic (PLEG): container finished" podID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerID="02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c" exitCode=0 Oct 30 15:26:15.851452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.851408 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" Oct 30 15:26:15.851980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.851457 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" event={"ID":"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e","Type":"ContainerDied","Data":"02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c"} Oct 30 15:26:15.851980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.851523 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq" event={"ID":"5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e","Type":"ContainerDied","Data":"a1b603cd614c732beaaa72f289d5ba803d0253d4ec05b7d4a9244628a19f1930"} Oct 30 15:26:15.851980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.851556 6554 scope.go:122] "RemoveContainer" containerID="02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c" Oct 30 15:26:15.862479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.862463 6554 scope.go:122] "RemoveContainer" containerID="02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c" Oct 30 15:26:15.862711 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:15.862692 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c\": container with ID starting with 02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c not found: ID does not exist" containerID="02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c" Oct 30 15:26:15.862760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.862718 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c"} err="failed to get container status \"02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c\": rpc error: code = NotFound desc = could not find container \"02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c\": container with ID starting with 02d71de026114bb75fa06fedf5c86d7e8a0c541250b9d79f38473b7062d25d3c not found: ID does not exist" Oct 30 15:26:15.909524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.909501 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq"] Oct 30 15:26:15.922506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:15.922484 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a9275a-5cd57cdff-7whsq"] Oct 30 15:26:16.156135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:16.156111 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" path="/var/lib/kubelet/pods/55618dab-3ed3-40bc-8f0e-68e4bc6893a0/volumes" Oct 30 15:26:16.156522 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:16.156507 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" path="/var/lib/kubelet/pods/5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e/volumes" Oct 30 15:26:22.677335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:22.677299 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm"] Oct 30 15:26:22.718019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:22.717991 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws"] Oct 30 15:26:27.677612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:27.677572 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerName="payload-processing" containerID="cri-o://c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a" gracePeriod=25 Oct 30 15:26:27.718366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:27.718329 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" podUID="a89b90da-c993-43e2-961e-9add1860eec6" containerName="payload-pre-processing" containerID="cri-o://c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd" gracePeriod=25 Oct 30 15:26:35.176509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:35.176466 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:35.176957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:35.176698 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" podUID="a89b90da-c993-43e2-961e-9add1860eec6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:37.542868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.542833 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr"] Oct 30 15:26:37.543758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.543734 6554 cpu_manager.go:414] "containerMap: removing container" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" Oct 30 15:26:37.543758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.543759 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" Oct 30 15:26:37.543941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.543801 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5b8a4ed0-e17a-437a-9eba-91356b584128" containerName="payload-processing" Oct 30 15:26:37.543941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.543809 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b8a4ed0-e17a-437a-9eba-91356b584128" containerName="payload-processing" Oct 30 15:26:37.543941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.543839 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerName="payload-pre-processing" Oct 30 15:26:37.543941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.543846 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerName="payload-pre-processing" Oct 30 15:26:37.544145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.544047 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b8a4ed0-e17a-437a-9eba-91356b584128" containerName="payload-processing" Oct 30 15:26:37.544145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.544067 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb4d2c0-31af-4ce0-b10c-ce7c8a33321e" containerName="payload-pre-processing" Oct 30 15:26:37.544145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.544088 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="55618dab-3ed3-40bc-8f0e-68e4bc6893a0" containerName="istio-proxy" Oct 30 15:26:37.549777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.549752 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.575174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.575149 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr"] Oct 30 15:26:37.582862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.582835 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-5a5948-openshift-default-dockercfg-zj9cx\"" Oct 30 15:26:37.609252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609223 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609394 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd8z2\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609597 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609649 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609706 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609799 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609871 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.609944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.609925 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.610051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.610031 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.710725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710698 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd8z2\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.710839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710797 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.710839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710822 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.710951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710845 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.710951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710894 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.710951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710929 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.710978 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711021 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711082 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711240 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711287 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711564 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711652 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711606 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.711953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.711928 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.713185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.713149 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.713665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.713643 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.754388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.754312 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd8z2\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.758408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.758335 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token\") pod \"e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:37.879184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:37.879110 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:38.215652 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:38.215626 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr"] Oct 30 15:26:38.216833 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:26:38.216805 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a67a0ab_124f_412e_b5ee_b4bdadb5613d.slice/crio-996a94d1ec3ed4e42c8f9803348dc9176f96f3496de366dd6d988a528af0bc6b WatchSource:0}: Error finding container 996a94d1ec3ed4e42c8f9803348dc9176f96f3496de366dd6d988a528af0bc6b: Status 404 returned error can't find the container with id 996a94d1ec3ed4e42c8f9803348dc9176f96f3496de366dd6d988a528af0bc6b Oct 30 15:26:38.219995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:38.219955 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:26:38.220075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:38.220038 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:26:38.220116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:38.220080 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:26:39.025417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:39.025380 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" event={"ID":"1a67a0ab-124f-412e-b5ee-b4bdadb5613d","Type":"ContainerStarted","Data":"b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f"} Oct 30 15:26:39.025818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:39.025436 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" event={"ID":"1a67a0ab-124f-412e-b5ee-b4bdadb5613d","Type":"ContainerStarted","Data":"996a94d1ec3ed4e42c8f9803348dc9176f96f3496de366dd6d988a528af0bc6b"} Oct 30 15:26:39.088521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:39.088474 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podStartSLOduration=2.088460485 podStartE2EDuration="2.088460485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:26:39.086750286 +0000 UTC m=+2303.045791728" watchObservedRunningTime="2026-10-30 15:26:39.088460485 +0000 UTC m=+2303.047501890" Oct 30 15:26:39.223129 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:39.223082 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:26:39.879883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:39.879852 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:39.885775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:39.885749 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:40.032870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:40.032834 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:40.033927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:40.033905 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:26:45.177103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:45.177069 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" podUID="a89b90da-c993-43e2-961e-9add1860eec6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:45.177528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:45.177293 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:26:48.004179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.004150 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:26:48.038417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.038392 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:26:48.090153 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.090119 6554 generic.go:356] "Generic (PLEG): container finished" podID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerID="c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a" exitCode=0 Oct 30 15:26:48.090309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.090167 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" Oct 30 15:26:48.090309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.090183 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" event={"ID":"ea3ae15e-034a-4ed7-a456-744a1066bde7","Type":"ContainerDied","Data":"c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a"} Oct 30 15:26:48.090309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.090242 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm" event={"ID":"ea3ae15e-034a-4ed7-a456-744a1066bde7","Type":"ContainerDied","Data":"92c3aa61c23ca520229898460bef8c58f7a9c5aef2c441e954fd4f5849f9678e"} Oct 30 15:26:48.090309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.090270 6554 scope.go:122] "RemoveContainer" containerID="c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a" Oct 30 15:26:48.092413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.092166 6554 generic.go:356] "Generic (PLEG): container finished" podID="a89b90da-c993-43e2-961e-9add1860eec6" containerID="c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd" exitCode=0 Oct 30 15:26:48.092413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.092220 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" Oct 30 15:26:48.092413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.092270 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" event={"ID":"a89b90da-c993-43e2-961e-9add1860eec6","Type":"ContainerDied","Data":"c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd"} Oct 30 15:26:48.092413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.092310 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws" event={"ID":"a89b90da-c993-43e2-961e-9add1860eec6","Type":"ContainerDied","Data":"20ad3cac087058c532d486ee9f9bc36f98e3c2514b5fde0477a2c1b85afe60f3"} Oct 30 15:26:48.111730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.111705 6554 scope.go:122] "RemoveContainer" containerID="c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a" Oct 30 15:26:48.112661 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:48.112633 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a\": container with ID starting with c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a not found: ID does not exist" containerID="c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a" Oct 30 15:26:48.112779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.112668 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a"} err="failed to get container status \"c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a\": rpc error: code = NotFound desc = could not find container \"c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a\": container with ID starting with c02df238a9d85243869018cffa61f71bb61c431b2050a04cddc91e364336d78a not found: ID does not exist" Oct 30 15:26:48.112779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.112692 6554 scope.go:122] "RemoveContainer" containerID="c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd" Oct 30 15:26:48.132870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.132831 6554 scope.go:122] "RemoveContainer" containerID="c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd" Oct 30 15:26:48.133167 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:48.133146 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd\": container with ID starting with c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd not found: ID does not exist" containerID="c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd" Oct 30 15:26:48.133254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.133174 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd"} err="failed to get container status \"c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd\": rpc error: code = NotFound desc = could not find container \"c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd\": container with ID starting with c9d508f56942f2678fb0c0f1df76b84113fa10f95dd84cb95b4e84a883b073bd not found: ID does not exist" Oct 30 15:26:48.150943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.150919 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume\") pod \"a89b90da-c993-43e2-961e-9add1860eec6\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " Oct 30 15:26:48.151053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.150961 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume\") pod \"ea3ae15e-034a-4ed7-a456-744a1066bde7\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " Oct 30 15:26:48.151120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.151099 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w\" (UniqueName: \"kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w\") pod \"a89b90da-c993-43e2-961e-9add1860eec6\" (UID: \"a89b90da-c993-43e2-961e-9add1860eec6\") " Oct 30 15:26:48.151182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.151126 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r\" (UniqueName: \"kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r\") pod \"ea3ae15e-034a-4ed7-a456-744a1066bde7\" (UID: \"ea3ae15e-034a-4ed7-a456-744a1066bde7\") " Oct 30 15:26:48.151335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.151310 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume" pod "a89b90da-c993-43e2-961e-9add1860eec6" (UID: "a89b90da-c993-43e2-961e-9add1860eec6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:48.151460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.151438 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume" pod "ea3ae15e-034a-4ed7-a456-744a1066bde7" (UID: "ea3ae15e-034a-4ed7-a456-744a1066bde7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:48.151646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.151629 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a89b90da-c993-43e2-961e-9add1860eec6-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:48.151708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.151651 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea3ae15e-034a-4ed7-a456-744a1066bde7-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:48.153112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.153088 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w" pod "a89b90da-c993-43e2-961e-9add1860eec6" (UID: "a89b90da-c993-43e2-961e-9add1860eec6"). InnerVolumeSpecName "kube-api-access-4kw2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:48.153303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.153286 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r" pod "ea3ae15e-034a-4ed7-a456-744a1066bde7" (UID: "ea3ae15e-034a-4ed7-a456-744a1066bde7"). InnerVolumeSpecName "kube-api-access-qtl7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:48.253324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.253266 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kw2w\" (UniqueName: \"kubernetes.io/projected/a89b90da-c993-43e2-961e-9add1860eec6-kube-api-access-4kw2w\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:48.253324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.253293 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtl7r\" (UniqueName: \"kubernetes.io/projected/ea3ae15e-034a-4ed7-a456-744a1066bde7-kube-api-access-qtl7r\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:48.436852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.436822 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm"] Oct 30 15:26:48.452396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.452368 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7c5235-59d978ccd4-2pjbm"] Oct 30 15:26:48.479075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.479039 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws"] Oct 30 15:26:48.483337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:48.483312 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7c5235-5f65c98c5-86hws"] Oct 30 15:26:50.169006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:50.168967 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a89b90da-c993-43e2-961e-9add1860eec6" path="/var/lib/kubelet/pods/a89b90da-c993-43e2-961e-9add1860eec6/volumes" Oct 30 15:26:50.171447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:50.171421 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" path="/var/lib/kubelet/pods/ea3ae15e-034a-4ed7-a456-744a1066bde7/volumes" Oct 30 15:26:57.087298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.087268 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk"] Oct 30 15:26:57.087912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.087543 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" podUID="263d797f-de64-4beb-8fb0-84811b1a22b7" containerName="maas-api" containerID="cri-o://f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c" gracePeriod=30 Oct 30 15:26:57.412330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.411896 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:26:57.490489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.490451 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs\") pod \"263d797f-de64-4beb-8fb0-84811b1a22b7\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " Oct 30 15:26:57.490688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.490517 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls\") pod \"263d797f-de64-4beb-8fb0-84811b1a22b7\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " Oct 30 15:26:57.490688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.490565 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca\") pod \"263d797f-de64-4beb-8fb0-84811b1a22b7\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " Oct 30 15:26:57.490825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.490788 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr\" (UniqueName: \"kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr\") pod \"263d797f-de64-4beb-8fb0-84811b1a22b7\" (UID: \"263d797f-de64-4beb-8fb0-84811b1a22b7\") " Oct 30 15:26:57.490990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.490962 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca" pod "263d797f-de64-4beb-8fb0-84811b1a22b7" (UID: "263d797f-de64-4beb-8fb0-84811b1a22b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:26:57.491435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.491410 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263d797f-de64-4beb-8fb0-84811b1a22b7-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:57.494097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.493452 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr" pod "263d797f-de64-4beb-8fb0-84811b1a22b7" (UID: "263d797f-de64-4beb-8fb0-84811b1a22b7"). InnerVolumeSpecName "kube-api-access-kvfbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:26:57.494097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.493718 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls" pod "263d797f-de64-4beb-8fb0-84811b1a22b7" (UID: "263d797f-de64-4beb-8fb0-84811b1a22b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:57.495986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.495956 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs" pod "263d797f-de64-4beb-8fb0-84811b1a22b7" (UID: "263d797f-de64-4beb-8fb0-84811b1a22b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:26:57.595301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.595161 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfbr\" (UniqueName: \"kubernetes.io/projected/263d797f-de64-4beb-8fb0-84811b1a22b7-kube-api-access-kvfbr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:57.595301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.595200 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:57.595301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:57.595215 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263d797f-de64-4beb-8fb0-84811b1a22b7-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:26:58.189877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.189820 6554 generic.go:356] "Generic (PLEG): container finished" podID="263d797f-de64-4beb-8fb0-84811b1a22b7" containerID="f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c" exitCode=0 Oct 30 15:26:58.190394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.189905 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" Oct 30 15:26:58.190394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.190126 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" event={"ID":"263d797f-de64-4beb-8fb0-84811b1a22b7","Type":"ContainerDied","Data":"f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c"} Oct 30 15:26:58.190394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.190179 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk" event={"ID":"263d797f-de64-4beb-8fb0-84811b1a22b7","Type":"ContainerDied","Data":"2500dcc3cba6e3b45004ce3e8c7e2acec8c9a8939d752c89e6a896b17fae0f7d"} Oct 30 15:26:58.190394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.190209 6554 scope.go:122] "RemoveContainer" containerID="f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c" Oct 30 15:26:58.202967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.202940 6554 scope.go:122] "RemoveContainer" containerID="f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c" Oct 30 15:26:58.203335 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:26:58.203302 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c\": container with ID starting with f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c not found: ID does not exist" containerID="f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c" Oct 30 15:26:58.203503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.203361 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c"} err="failed to get container status \"f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c\": rpc error: code = NotFound desc = could not find container \"f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c\": container with ID starting with f370ea7b04c8ac8a5a48dfc9a7c4a184172e05e4c397c592996bbac4cdc70f9c not found: ID does not exist" Oct 30 15:26:58.246989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.246963 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk"] Oct 30 15:26:58.260435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:26:58.260412 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7c5235-7d4c4c4999-s4qkk"] Oct 30 15:27:00.172688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:00.172650 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="263d797f-de64-4beb-8fb0-84811b1a22b7" path="/var/lib/kubelet/pods/263d797f-de64-4beb-8fb0-84811b1a22b7/volumes" Oct 30 15:27:02.045040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045006 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7"] Oct 30 15:27:02.045808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045789 6554 cpu_manager.go:414] "containerMap: removing container" podUID="263d797f-de64-4beb-8fb0-84811b1a22b7" containerName="maas-api" Oct 30 15:27:02.045862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045810 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="263d797f-de64-4beb-8fb0-84811b1a22b7" containerName="maas-api" Oct 30 15:27:02.045862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045825 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerName="payload-processing" Oct 30 15:27:02.045862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045830 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerName="payload-processing" Oct 30 15:27:02.045862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045849 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a89b90da-c993-43e2-961e-9add1860eec6" containerName="payload-pre-processing" Oct 30 15:27:02.045862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045855 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a89b90da-c993-43e2-961e-9add1860eec6" containerName="payload-pre-processing" Oct 30 15:27:02.046017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045956 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3ae15e-034a-4ed7-a456-744a1066bde7" containerName="payload-processing" Oct 30 15:27:02.046017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045968 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a89b90da-c993-43e2-961e-9add1860eec6" containerName="payload-pre-processing" Oct 30 15:27:02.046017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.045983 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="263d797f-de64-4beb-8fb0-84811b1a22b7" containerName="maas-api" Oct 30 15:27:02.051387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.051345 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.075770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.075737 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7"] Oct 30 15:27:02.085636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.085608 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-7024f9-serving-cert\"" Oct 30 15:27:02.147122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.147093 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.147295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.147197 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.147295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.147242 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.147635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.147616 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwclr\" (UniqueName: \"kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.248578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.248538 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.248749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.248619 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.248749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.248658 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.248879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.248825 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwclr\" (UniqueName: \"kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.249436 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.249409 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.251080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.251061 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.251146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.251119 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.283488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.283450 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwclr\" (UniqueName: \"kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr\") pod \"maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.366844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.366761 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:02.698078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.698044 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7"] Oct 30 15:27:02.698413 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:27:02.698390 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceb578_984d_43cd_8797_50a286dea383.slice/crio-affde0ab85fd28ec2daf60169a6e793349ba5bd080330b48b07d0cd070ccc116 WatchSource:0}: Error finding container affde0ab85fd28ec2daf60169a6e793349ba5bd080330b48b07d0cd070ccc116: Status 404 returned error can't find the container with id affde0ab85fd28ec2daf60169a6e793349ba5bd080330b48b07d0cd070ccc116 Oct 30 15:27:02.699778 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:02.699761 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:27:03.230201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:03.230166 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" event={"ID":"f5ceb578-984d-43cd-8797-50a286dea383","Type":"ContainerStarted","Data":"b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2"} Oct 30 15:27:03.230653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:03.230228 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" event={"ID":"f5ceb578-984d-43cd-8797-50a286dea383","Type":"ContainerStarted","Data":"affde0ab85fd28ec2daf60169a6e793349ba5bd080330b48b07d0cd070ccc116"} Oct 30 15:27:03.230653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:03.230274 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:03.277544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:03.277493 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" podStartSLOduration=0.931119946 podStartE2EDuration="1.277479706s" totalImagesPullingTime="346.35976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:03.275451265 +0000 UTC m=+2327.234492680" watchObservedRunningTime="2026-10-30 15:27:03.277479706 +0000 UTC m=+2327.236521110" Oct 30 15:27:09.202408 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:09.202367 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:27:09.247996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:09.247973 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:27:13.933585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:13.933540 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s"] Oct 30 15:27:13.938037 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:13.938015 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:13.963902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:13.963874 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s"] Oct 30 15:27:14.087167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087139 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087168 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087200 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087305 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087403 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbpz\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087469 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087559 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087586 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.087615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.087606 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189214 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189244 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189261 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189312 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189328 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189393 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189444 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189546 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189508 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbpz\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189614 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189668 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.189863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189760 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.190021 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.189975 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.190081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.190028 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.190135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.190094 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.191718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.191699 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.191968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.191949 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.214815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.214791 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.214940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.214900 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbpz\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz\") pod \"e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.252750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.252730 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:14.581768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.581742 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s"] Oct 30 15:27:14.582615 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:27:14.582592 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba1a5c4_6120_4e47_83ae_f23cd063b1f2.slice/crio-9c3b7ba46d456bbc7462140b6b996d9c9b94a35b69d52854189c8d1ee05267e6 WatchSource:0}: Error finding container 9c3b7ba46d456bbc7462140b6b996d9c9b94a35b69d52854189c8d1ee05267e6: Status 404 returned error can't find the container with id 9c3b7ba46d456bbc7462140b6b996d9c9b94a35b69d52854189c8d1ee05267e6 Oct 30 15:27:14.585551 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.585517 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:27:14.585647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.585594 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:27:14.585647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:14.585625 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:27:15.060105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.060073 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7"] Oct 30 15:27:15.060630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.060312 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" podUID="16503d59-114c-408a-9ea1-a60f4f6e663a" containerName="maas-api" containerID="cri-o://aa8ef84da091d339500fbbd3f739797d6be54bb8f285efc1bec4bb0abefeb7fa" gracePeriod=30 Oct 30 15:27:15.358724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.357309 6554 generic.go:356] "Generic (PLEG): container finished" podID="16503d59-114c-408a-9ea1-a60f4f6e663a" containerID="aa8ef84da091d339500fbbd3f739797d6be54bb8f285efc1bec4bb0abefeb7fa" exitCode=0 Oct 30 15:27:15.358724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.357390 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" event={"ID":"16503d59-114c-408a-9ea1-a60f4f6e663a","Type":"ContainerDied","Data":"aa8ef84da091d339500fbbd3f739797d6be54bb8f285efc1bec4bb0abefeb7fa"} Oct 30 15:27:15.360393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.359552 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" event={"ID":"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2","Type":"ContainerStarted","Data":"22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc"} Oct 30 15:27:15.360393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.359604 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" event={"ID":"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2","Type":"ContainerStarted","Data":"9c3b7ba46d456bbc7462140b6b996d9c9b94a35b69d52854189c8d1ee05267e6"} Oct 30 15:27:15.405537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.405510 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:27:15.409550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.409491 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podStartSLOduration=2.409473699 podStartE2EDuration="2.409473699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:15.409090057 +0000 UTC m=+2339.368131475" watchObservedRunningTime="2026-10-30 15:27:15.409473699 +0000 UTC m=+2339.368515104" Oct 30 15:27:15.464377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.458895 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs\") pod \"16503d59-114c-408a-9ea1-a60f4f6e663a\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " Oct 30 15:27:15.464758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.464730 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\") pod \"16503d59-114c-408a-9ea1-a60f4f6e663a\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " Oct 30 15:27:15.464926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.464907 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g\" (UniqueName: \"kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g\") pod \"16503d59-114c-408a-9ea1-a60f4f6e663a\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " Oct 30 15:27:15.465085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.465067 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca\") pod \"16503d59-114c-408a-9ea1-a60f4f6e663a\" (UID: \"16503d59-114c-408a-9ea1-a60f4f6e663a\") " Oct 30 15:27:15.465363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.465322 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs" pod "16503d59-114c-408a-9ea1-a60f4f6e663a" (UID: "16503d59-114c-408a-9ea1-a60f4f6e663a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:15.488372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.466064 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:15.488787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.467945 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca" pod "16503d59-114c-408a-9ea1-a60f4f6e663a" (UID: "16503d59-114c-408a-9ea1-a60f4f6e663a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:15.488897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.482674 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls" pod "16503d59-114c-408a-9ea1-a60f4f6e663a" (UID: "16503d59-114c-408a-9ea1-a60f4f6e663a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:15.500371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.497944 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g" pod "16503d59-114c-408a-9ea1-a60f4f6e663a" (UID: "16503d59-114c-408a-9ea1-a60f4f6e663a"). InnerVolumeSpecName "kube-api-access-pf47g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:15.590820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.590778 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16503d59-114c-408a-9ea1-a60f4f6e663a-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:15.590820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.590821 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf47g\" (UniqueName: \"kubernetes.io/projected/16503d59-114c-408a-9ea1-a60f4f6e663a-kube-api-access-pf47g\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:15.591025 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:15.590839 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16503d59-114c-408a-9ea1-a60f4f6e663a-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:16.260531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.260502 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:16.280227 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.280187 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 15:27:16.280488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.280266 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 15:27:16.388634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.388606 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" Oct 30 15:27:16.388634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.388615 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7" event={"ID":"16503d59-114c-408a-9ea1-a60f4f6e663a","Type":"ContainerDied","Data":"db2429c2cf41d8f01d3cc59879dcb64879bb03978148efffb35157784a210229"} Oct 30 15:27:16.388871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.388678 6554 scope.go:122] "RemoveContainer" containerID="aa8ef84da091d339500fbbd3f739797d6be54bb8f285efc1bec4bb0abefeb7fa" Oct 30 15:27:16.448823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.448764 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7"] Oct 30 15:27:16.479699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:16.479666 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-3acdc3-76b484b8f6-n65z7"] Oct 30 15:27:17.253089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:17.253055 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 15:27:17.253240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:17.253104 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 15:27:18.175372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:18.170526 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16503d59-114c-408a-9ea1-a60f4f6e663a" path="/var/lib/kubelet/pods/16503d59-114c-408a-9ea1-a60f4f6e663a/volumes" Oct 30 15:27:18.253369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:18.253318 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 15:27:18.253566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:18.253414 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 15:27:19.257373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:19.256645 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 15:27:19.257373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:19.256706 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 15:27:20.253289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:20.253258 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 15:27:20.253525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:20.253307 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 15:27:21.253636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:21.253603 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 15:27:21.254051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:21.253663 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 15:27:22.257960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:22.257932 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:22.258493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:22.258317 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:22.258909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:22.258891 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:27:23.794198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:23.794162 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk"] Oct 30 15:27:23.794741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:23.794403 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" containerID="cri-o://87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d" gracePeriod=30 Oct 30 15:27:25.227498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:25.227463 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf"] Oct 30 15:27:25.260703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:25.260676 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g"] Oct 30 15:27:29.513393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.513363 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:27:29.563979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.563949 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564043 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564096 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564193 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564215 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564244 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564273 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564369 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.564557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.564431 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs\") pod \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\" (UID: \"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc\") " Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565132 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565198 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565265 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565395 6554 generic.go:356] "Generic (PLEG): container finished" podID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerID="87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d" exitCode=0 Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565512 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" event={"ID":"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc","Type":"ContainerDied","Data":"87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d"} Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565590 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565608 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" event={"ID":"7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc","Type":"ContainerDied","Data":"b9833a3ae97c1dd1f9afc6ad735afc20ada91bb31cb953647becb0a8c2654244"} Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565641 6554 scope.go:122] "RemoveContainer" containerID="87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d" Oct 30 15:27:29.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.565685 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk" Oct 30 15:27:29.566825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.566616 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.566825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.566742 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "kube-api-access-br9nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:29.567030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.567007 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:29.567695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.567420 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:27:29.567695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.567642 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy" pod "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" (UID: "7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:27:29.600742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.600715 6554 scope.go:122] "RemoveContainer" containerID="814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6" Oct 30 15:27:29.611051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.611033 6554 scope.go:122] "RemoveContainer" containerID="87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d" Oct 30 15:27:29.611401 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:29.611338 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d\": container with ID starting with 87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d not found: ID does not exist" containerID="87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d" Oct 30 15:27:29.611470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.611412 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d"} err="failed to get container status \"87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d\": rpc error: code = NotFound desc = could not find container \"87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d\": container with ID starting with 87f68ed18112af621573b4564b1d67e3e5245c5bd897744f5363ee5b7d80196d not found: ID does not exist" Oct 30 15:27:29.611470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.611435 6554 scope.go:122] "RemoveContainer" containerID="814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6" Oct 30 15:27:29.611701 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:29.611684 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6\": container with ID starting with 814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6 not found: ID does not exist" containerID="814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6" Oct 30 15:27:29.611754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.611706 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6"} err="failed to get container status \"814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6\": rpc error: code = NotFound desc = could not find container \"814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6\": container with ID starting with 814cdd441ffcd02d8298c23519260ce7abec8fba6503dd13cc33cce4723c40b6 not found: ID does not exist" Oct 30 15:27:29.666877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666852 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.666877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666877 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666889 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666901 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666913 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br9nt\" (UniqueName: \"kubernetes.io/projected/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-kube-api-access-br9nt\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666924 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666934 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666942 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.667010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.666949 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:29.916664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.916630 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk"] Oct 30 15:27:29.930448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:29.930422 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3acdc3-openshift-default-686fd5f449-bq8gk"] Oct 30 15:27:30.156199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:30.156172 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" path="/var/lib/kubelet/pods/7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc/volumes" Oct 30 15:27:30.228517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:30.228480 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerName="payload-processing" containerID="cri-o://3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f" gracePeriod=25 Oct 30 15:27:30.261421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:30.261391 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerName="payload-pre-processing" containerID="cri-o://531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a" gracePeriod=25 Oct 30 15:27:31.140316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:31.140277 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:27:31.140316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:31.140302 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:27:37.285417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.285385 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s"] Oct 30 15:27:37.285902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.285579 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" podUID="66877f18-0b78-4635-b53d-e5d7fc3e6a62" containerName="maas-api" containerID="cri-o://6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b" gracePeriod=30 Oct 30 15:27:37.589431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.588670 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:27:37.634827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.634791 6554 generic.go:356] "Generic (PLEG): container finished" podID="66877f18-0b78-4635-b53d-e5d7fc3e6a62" containerID="6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b" exitCode=0 Oct 30 15:27:37.635030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.634885 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" event={"ID":"66877f18-0b78-4635-b53d-e5d7fc3e6a62","Type":"ContainerDied","Data":"6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b"} Oct 30 15:27:37.635030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.634932 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" event={"ID":"66877f18-0b78-4635-b53d-e5d7fc3e6a62","Type":"ContainerDied","Data":"72c1ae0885abe4a49ae56a1ee90a5e613076931f98c8a7485e93476bdd1eaf61"} Oct 30 15:27:37.635030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.634955 6554 scope.go:122] "RemoveContainer" containerID="6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b" Oct 30 15:27:37.635205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.635101 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s" Oct 30 15:27:37.647327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.647301 6554 scope.go:122] "RemoveContainer" containerID="6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b" Oct 30 15:27:37.647712 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:37.647684 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b\": container with ID starting with 6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b not found: ID does not exist" containerID="6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b" Oct 30 15:27:37.647795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.647723 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b"} err="failed to get container status \"6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b\": rpc error: code = NotFound desc = could not find container \"6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b\": container with ID starting with 6ad28ea8f93e037db1dc904a62235cbd05ba3267d615aac7e7e2d9bc6e501b2b not found: ID does not exist" Oct 30 15:27:37.708113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.708085 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca\") pod \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " Oct 30 15:27:37.708300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.708144 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs\") pod \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " Oct 30 15:27:37.708300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.708201 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588\" (UniqueName: \"kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588\") pod \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " Oct 30 15:27:37.708434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.708378 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls\") pod \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\" (UID: \"66877f18-0b78-4635-b53d-e5d7fc3e6a62\") " Oct 30 15:27:37.708528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.708495 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca" pod "66877f18-0b78-4635-b53d-e5d7fc3e6a62" (UID: "66877f18-0b78-4635-b53d-e5d7fc3e6a62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:37.709007 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.708984 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/66877f18-0b78-4635-b53d-e5d7fc3e6a62-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:37.710708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.710684 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls" pod "66877f18-0b78-4635-b53d-e5d7fc3e6a62" (UID: "66877f18-0b78-4635-b53d-e5d7fc3e6a62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:37.710905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.710887 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588" pod "66877f18-0b78-4635-b53d-e5d7fc3e6a62" (UID: "66877f18-0b78-4635-b53d-e5d7fc3e6a62"). InnerVolumeSpecName "kube-api-access-jh588". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:37.710991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.710911 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs" pod "66877f18-0b78-4635-b53d-e5d7fc3e6a62" (UID: "66877f18-0b78-4635-b53d-e5d7fc3e6a62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:27:37.822376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.822041 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:37.822376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.822078 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66877f18-0b78-4635-b53d-e5d7fc3e6a62-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:37.822376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:37.822092 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh588\" (UniqueName: \"kubernetes.io/projected/66877f18-0b78-4635-b53d-e5d7fc3e6a62-kube-api-access-jh588\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:38.000686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:38.000654 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s"] Oct 30 15:27:38.005846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:38.005814 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-859a8d-db9b5f6b4-8mp6s"] Oct 30 15:27:38.156276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:38.156235 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66877f18-0b78-4635-b53d-e5d7fc3e6a62" path="/var/lib/kubelet/pods/66877f18-0b78-4635-b53d-e5d7fc3e6a62/volumes" Oct 30 15:27:39.202285 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:39.202253 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:27:41.139787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.139745 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:27:41.139787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.139767 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:27:41.583522 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.583443 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n"] Oct 30 15:27:41.584272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584241 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" Oct 30 15:27:41.584272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584267 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" Oct 30 15:27:41.584503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584289 6554 cpu_manager.go:414] "containerMap: removing container" podUID="16503d59-114c-408a-9ea1-a60f4f6e663a" containerName="maas-api" Oct 30 15:27:41.584503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584297 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16503d59-114c-408a-9ea1-a60f4f6e663a" containerName="maas-api" Oct 30 15:27:41.584503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584317 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" Oct 30 15:27:41.584503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584326 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" Oct 30 15:27:41.584503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584378 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66877f18-0b78-4635-b53d-e5d7fc3e6a62" containerName="maas-api" Oct 30 15:27:41.584503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584384 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66877f18-0b78-4635-b53d-e5d7fc3e6a62" containerName="maas-api" Oct 30 15:27:41.584775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584512 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" Oct 30 15:27:41.584775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584522 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="16503d59-114c-408a-9ea1-a60f4f6e663a" containerName="maas-api" Oct 30 15:27:41.584775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584531 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7887332a-4ea2-4d7e-a89e-aab2ffe4a6fc" containerName="istio-proxy" Oct 30 15:27:41.584775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.584540 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66877f18-0b78-4635-b53d-e5d7fc3e6a62" containerName="maas-api" Oct 30 15:27:41.588011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.587992 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.604527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.604502 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-13009a-openshift-default-dockercfg-2gpnk\"" Oct 30 15:27:41.621821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.621794 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n"] Oct 30 15:27:41.666984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.666960 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.666999 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667030 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667186 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zf8\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667205 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667250 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667317 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667514 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.667597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.667560 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.768763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.768735 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zf8\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.768901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.768769 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.768901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.768818 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.768901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.768839 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.768940 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.768973 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769022 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769060 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769116 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769253 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769464 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769478 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769602 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.769866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.769848 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.771421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.771402 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.771710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.771674 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.793507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.793489 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.794014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.793992 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zf8\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8\") pod \"e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:41.902031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:41.902010 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:42.247863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.247832 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n"] Oct 30 15:27:42.248696 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:27:42.248669 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff59fa2_d93a_451c_9db0_b6c7ae616001.slice/crio-bff04b830e6f85d6ddae79b6e34c2f97473ab649ba9539b561167866b797f5bd WatchSource:0}: Error finding container bff04b830e6f85d6ddae79b6e34c2f97473ab649ba9539b561167866b797f5bd: Status 404 returned error can't find the container with id bff04b830e6f85d6ddae79b6e34c2f97473ab649ba9539b561167866b797f5bd Oct 30 15:27:42.251416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.251383 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:27:42.251520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.251448 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:27:42.251520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.251482 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:27:42.670268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.670228 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" event={"ID":"2ff59fa2-d93a-451c-9db0-b6c7ae616001","Type":"ContainerStarted","Data":"cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3"} Oct 30 15:27:42.670476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.670284 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" event={"ID":"2ff59fa2-d93a-451c-9db0-b6c7ae616001","Type":"ContainerStarted","Data":"bff04b830e6f85d6ddae79b6e34c2f97473ab649ba9539b561167866b797f5bd"} Oct 30 15:27:42.720101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.719527 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podStartSLOduration=1.7195083869999999 podStartE2EDuration="1.719508387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:27:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:27:42.717895397 +0000 UTC m=+2366.676936809" watchObservedRunningTime="2026-10-30 15:27:42.719508387 +0000 UTC m=+2366.678549793" Oct 30 15:27:42.902460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:42.902427 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:43.908658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:43.908040 6554 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 15:27:43.908658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:43.908116 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 15:27:44.902999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:44.902963 6554 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 15:27:44.903198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:44.903020 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 15:27:45.903383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:45.903333 6554 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 15:27:45.903794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:45.903405 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 15:27:46.902305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:46.902273 6554 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 15:27:46.902505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:46.902321 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 15:27:47.902684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:47.902647 6554 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 15:27:47.903079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:47.902703 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 15:27:48.908073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:48.908041 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:48.908548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:48.908529 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:48.909193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:48.909175 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:27:50.360316 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:50.360286 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a9e8d_c6c2_4089_8250_51dee9f0e38e.slice/crio-conmon-531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a9e8d_c6c2_4089_8250_51dee9f0e38e.slice/crio-531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:27:50.563185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.563162 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:27:50.566759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.566737 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:27:50.737220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.737192 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp\" (UniqueName: \"kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp\") pod \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " Oct 30 15:27:50.737423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.737294 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume\") pod \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\" (UID: \"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e\") " Oct 30 15:27:50.737423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.737390 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume\") pod \"e5dae30e-ec65-427e-80a9-6c72cb470a49\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " Oct 30 15:27:50.737423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.737423 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm\" (UniqueName: \"kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm\") pod \"e5dae30e-ec65-427e-80a9-6c72cb470a49\" (UID: \"e5dae30e-ec65-427e-80a9-6c72cb470a49\") " Oct 30 15:27:50.737687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.737660 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume" pod "5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" (UID: "5a0a9e8d-c6c2-4089-8250-51dee9f0e38e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:50.737759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.737743 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume" pod "e5dae30e-ec65-427e-80a9-6c72cb470a49" (UID: "e5dae30e-ec65-427e-80a9-6c72cb470a49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:27:50.738039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.738022 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5dae30e-ec65-427e-80a9-6c72cb470a49-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:50.738039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.738040 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:50.739455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.739435 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp" pod "5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" (UID: "5a0a9e8d-c6c2-4089-8250-51dee9f0e38e"). InnerVolumeSpecName "kube-api-access-twnlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:50.739566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.739516 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm" pod "e5dae30e-ec65-427e-80a9-6c72cb470a49" (UID: "e5dae30e-ec65-427e-80a9-6c72cb470a49"). InnerVolumeSpecName "kube-api-access-zt4sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:27:50.775423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.775338 6554 generic.go:356] "Generic (PLEG): container finished" podID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerID="3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f" exitCode=0 Oct 30 15:27:50.775423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.775383 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" event={"ID":"e5dae30e-ec65-427e-80a9-6c72cb470a49","Type":"ContainerDied","Data":"3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f"} Oct 30 15:27:50.775423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.775398 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" Oct 30 15:27:50.775656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.775443 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf" event={"ID":"e5dae30e-ec65-427e-80a9-6c72cb470a49","Type":"ContainerDied","Data":"f16c4e932b53c1ef8c7d7b52bb6fd014f590f59d06f78e3fd67cfd2bd69474a5"} Oct 30 15:27:50.775656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.775466 6554 scope.go:122] "RemoveContainer" containerID="3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f" Oct 30 15:27:50.776855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.776829 6554 generic.go:356] "Generic (PLEG): container finished" podID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerID="531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a" exitCode=0 Oct 30 15:27:50.776979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.776857 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" event={"ID":"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e","Type":"ContainerDied","Data":"531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a"} Oct 30 15:27:50.776979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.776885 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" Oct 30 15:27:50.776979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.776893 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g" event={"ID":"5a0a9e8d-c6c2-4089-8250-51dee9f0e38e","Type":"ContainerDied","Data":"27c195418eb1bacf964e3e6a9fafffda3834bc9a6e26c136ba8c4c3dd2da46ad"} Oct 30 15:27:50.789570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.787920 6554 scope.go:122] "RemoveContainer" containerID="3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f" Oct 30 15:27:50.789570 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:50.788184 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f\": container with ID starting with 3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f not found: ID does not exist" containerID="3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f" Oct 30 15:27:50.789570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.788210 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f"} err="failed to get container status \"3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f\": rpc error: code = NotFound desc = could not find container \"3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f\": container with ID starting with 3f8f63413d03d1577cc11b61efba7776d2308610f8378dd4fcc7b6411db5a68f not found: ID does not exist" Oct 30 15:27:50.789570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.788232 6554 scope.go:122] "RemoveContainer" containerID="531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a" Oct 30 15:27:50.800985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.800954 6554 scope.go:122] "RemoveContainer" containerID="531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a" Oct 30 15:27:50.801271 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:27:50.801254 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a\": container with ID starting with 531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a not found: ID does not exist" containerID="531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a" Oct 30 15:27:50.801345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.801276 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a"} err="failed to get container status \"531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a\": rpc error: code = NotFound desc = could not find container \"531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a\": container with ID starting with 531b3c0f4a1eaf6035a067da125011e3b90023a9d2b278d597c926906d8e567a not found: ID does not exist" Oct 30 15:27:50.832664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.832633 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf"] Oct 30 15:27:50.837413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.837392 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-859a8d-57d57b6b7d-r5mrf"] Oct 30 15:27:50.838911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.838888 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt4sm\" (UniqueName: \"kubernetes.io/projected/e5dae30e-ec65-427e-80a9-6c72cb470a49-kube-api-access-zt4sm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:50.839003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.838913 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twnlp\" (UniqueName: \"kubernetes.io/projected/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e-kube-api-access-twnlp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:27:50.875265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.875229 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g"] Oct 30 15:27:50.882662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:50.882628 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-859a8d-5c76d9dcff-xhs8g"] Oct 30 15:27:52.160073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:52.160037 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" path="/var/lib/kubelet/pods/5a0a9e8d-c6c2-4089-8250-51dee9f0e38e/volumes" Oct 30 15:27:52.160667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:27:52.160646 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" path="/var/lib/kubelet/pods/e5dae30e-ec65-427e-80a9-6c72cb470a49/volumes" Oct 30 15:28:05.221717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.221676 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5"] Oct 30 15:28:05.222760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.222734 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerName="payload-pre-processing" Oct 30 15:28:05.222760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.222758 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerName="payload-pre-processing" Oct 30 15:28:05.222963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.222784 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerName="payload-processing" Oct 30 15:28:05.222963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.222793 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerName="payload-processing" Oct 30 15:28:05.222963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.222962 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0a9e8d-c6c2-4089-8250-51dee9f0e38e" containerName="payload-pre-processing" Oct 30 15:28:05.223129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.222974 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dae30e-ec65-427e-80a9-6c72cb470a49" containerName="payload-processing" Oct 30 15:28:05.228724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.228701 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.250097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.250070 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7aede6-dockercfg-twmmr\"" Oct 30 15:28:05.250578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.250558 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7aede6\"" Oct 30 15:28:05.269770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.269740 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5"] Oct 30 15:28:05.289776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.289696 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq"] Oct 30 15:28:05.295824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.295800 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.363239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.363202 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq"] Oct 30 15:28:05.364572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.364539 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.364824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.364805 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm996\" (UniqueName: \"kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.466106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.466062 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5254\" (UniqueName: \"kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254\") pod \"payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.466303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.466149 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.466303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.466294 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.467492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.467432 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.467754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.467732 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm996\" (UniqueName: \"kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.558498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.555220 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm996\" (UniqueName: \"kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996\") pod \"payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.558498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.555614 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:05.571518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.570056 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5254\" (UniqueName: \"kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254\") pod \"payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.571518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.570127 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.571518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.571181 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.595999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.595965 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5254\" (UniqueName: \"kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254\") pod \"payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.745960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.745885 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:05.912998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:05.912952 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5"] Oct 30 15:28:06.112437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:06.112377 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq"] Oct 30 15:28:06.119797 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:28:06.119501 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode410084b_3ff4_4cb5_b190_50b5682837bd.slice/crio-b89d9feab9366fb420171e5fb3f26eefc7a2b76cce0abb17b49e9675f2119abd WatchSource:0}: Error finding container b89d9feab9366fb420171e5fb3f26eefc7a2b76cce0abb17b49e9675f2119abd: Status 404 returned error can't find the container with id b89d9feab9366fb420171e5fb3f26eefc7a2b76cce0abb17b49e9675f2119abd Oct 30 15:28:07.103050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.103003 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" event={"ID":"e410084b-3ff4-4cb5-b190-50b5682837bd","Type":"ContainerStarted","Data":"15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0"} Oct 30 15:28:07.103623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.103204 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" event={"ID":"e410084b-3ff4-4cb5-b190-50b5682837bd","Type":"ContainerStarted","Data":"b89d9feab9366fb420171e5fb3f26eefc7a2b76cce0abb17b49e9675f2119abd"} Oct 30 15:28:07.105419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.105391 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:07.136615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.136576 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" event={"ID":"3d8e017f-aed4-4f08-986f-433c5fdfc77d","Type":"ContainerStarted","Data":"41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a"} Oct 30 15:28:07.137727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.137704 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:07.137913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.137895 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" event={"ID":"3d8e017f-aed4-4f08-986f-433c5fdfc77d","Type":"ContainerStarted","Data":"be4964d01c1af0ee0b46e820d6a2a5869b89f5a97cdc6fb5bb26c92b79bcfc51"} Oct 30 15:28:07.236662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.236607 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" podStartSLOduration=2.236589447 podStartE2EDuration="2.236589447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:07.193514028 +0000 UTC m=+2391.152555444" watchObservedRunningTime="2026-10-30 15:28:07.236589447 +0000 UTC m=+2391.195630854" Oct 30 15:28:07.239534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:07.239494 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" podStartSLOduration=2.239481755 podStartE2EDuration="2.239481755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:07.235917697 +0000 UTC m=+2391.194959103" watchObservedRunningTime="2026-10-30 15:28:07.239481755 +0000 UTC m=+2391.198523159" Oct 30 15:28:09.206810 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:28:09.206774 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:28:11.287168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.287133 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2"] Oct 30 15:28:11.287919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.287883 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" podUID="dacbbb8e-285b-4e9a-be21-ca29117f6dfc" containerName="maas-api" containerID="cri-o://ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85" gracePeriod=30 Oct 30 15:28:11.800406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.798034 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:28:11.874373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.872289 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca\") pod \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " Oct 30 15:28:11.874373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.872378 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v\" (UniqueName: \"kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v\") pod \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " Oct 30 15:28:11.874373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.872436 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs\") pod \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " Oct 30 15:28:11.874373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.872515 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls\") pod \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\" (UID: \"dacbbb8e-285b-4e9a-be21-ca29117f6dfc\") " Oct 30 15:28:11.885372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.883539 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca" pod "dacbbb8e-285b-4e9a-be21-ca29117f6dfc" (UID: "dacbbb8e-285b-4e9a-be21-ca29117f6dfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:11.888939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.888902 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls" pod "dacbbb8e-285b-4e9a-be21-ca29117f6dfc" (UID: "dacbbb8e-285b-4e9a-be21-ca29117f6dfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:11.894806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.893871 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v" pod "dacbbb8e-285b-4e9a-be21-ca29117f6dfc" (UID: "dacbbb8e-285b-4e9a-be21-ca29117f6dfc"). InnerVolumeSpecName "kube-api-access-sgv6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:11.894806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.894191 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs" pod "dacbbb8e-285b-4e9a-be21-ca29117f6dfc" (UID: "dacbbb8e-285b-4e9a-be21-ca29117f6dfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:28:11.981690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.981646 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:11.981690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.981688 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgv6v\" (UniqueName: \"kubernetes.io/projected/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-kube-api-access-sgv6v\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:11.981967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.981705 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:11.981967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:11.981719 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dacbbb8e-285b-4e9a-be21-ca29117f6dfc-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:12.195089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.195046 6554 generic.go:356] "Generic (PLEG): container finished" podID="dacbbb8e-285b-4e9a-be21-ca29117f6dfc" containerID="ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85" exitCode=0 Oct 30 15:28:12.195258 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.195243 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" event={"ID":"dacbbb8e-285b-4e9a-be21-ca29117f6dfc","Type":"ContainerDied","Data":"ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85"} Oct 30 15:28:12.195338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.195322 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" event={"ID":"dacbbb8e-285b-4e9a-be21-ca29117f6dfc","Type":"ContainerDied","Data":"53213190c5e1db678c6f17ca515ed021b059b86575da6e706f25b44ac7675b79"} Oct 30 15:28:12.197668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.195389 6554 scope.go:122] "RemoveContainer" containerID="ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85" Oct 30 15:28:12.197889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.197870 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2" Oct 30 15:28:12.226808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.226777 6554 scope.go:122] "RemoveContainer" containerID="ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85" Oct 30 15:28:12.227362 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:28:12.227331 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85\": container with ID starting with ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85 not found: ID does not exist" containerID="ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85" Oct 30 15:28:12.227470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.227391 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85"} err="failed to get container status \"ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85\": rpc error: code = NotFound desc = could not find container \"ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85\": container with ID starting with ca009e35fb1622c0014bf1c015fae2e4e3d60033899b33ec62daa5083bde1d85 not found: ID does not exist" Oct 30 15:28:12.280023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.279967 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2"] Oct 30 15:28:12.283092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:12.283067 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-af2072-5b7c474cdb-4f4x2"] Oct 30 15:28:14.158388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:14.158321 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacbbb8e-285b-4e9a-be21-ca29117f6dfc" path="/var/lib/kubelet/pods/dacbbb8e-285b-4e9a-be21-ca29117f6dfc/volumes" Oct 30 15:28:17.229966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:17.229936 6554 generic.go:356] "Generic (PLEG): container finished" podID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerID="b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f" exitCode=0 Oct 30 15:28:17.230475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:17.230004 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" event={"ID":"1a67a0ab-124f-412e-b5ee-b4bdadb5613d","Type":"ContainerDied","Data":"b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f"} Oct 30 15:28:17.230475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:17.230309 6554 scope.go:122] "RemoveContainer" containerID="b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f" Oct 30 15:28:17.231768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:17.231738 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:17.231870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:17.231803 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:17.231870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:17.231851 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:18.238773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:18.238735 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" event={"ID":"1a67a0ab-124f-412e-b5ee-b4bdadb5613d","Type":"ContainerStarted","Data":"698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30"} Oct 30 15:28:18.879965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:18.879548 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:28:18.879965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:18.879670 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 15:28:18.879965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:18.879714 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 15:28:19.170725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:19.169509 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:28:19.174562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:19.174535 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:28:19.879580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:19.879544 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 15:28:19.880065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:19.879606 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 15:28:20.879976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:20.879940 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 15:28:20.880484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:20.880009 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 15:28:21.880059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:21.880022 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 15:28:21.880462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:21.880084 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 15:28:22.879781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:22.879743 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" start-of-body= Oct 30 15:28:22.879966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:22.879847 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.183:15021/healthz/ready\": dial tcp 10.134.0.183:15021: connect: connection refused" Oct 30 15:28:23.884130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:23.884099 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:28:23.884602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:23.884492 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:28:23.885080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:23.885061 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:28:32.341743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:32.341708 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n"] Oct 30 15:28:32.342174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:32.341962 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" containerID="cri-o://cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3" gracePeriod=30 Oct 30 15:28:33.811369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.809196 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz"] Oct 30 15:28:33.811369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.810395 6554 cpu_manager.go:414] "containerMap: removing container" podUID="dacbbb8e-285b-4e9a-be21-ca29117f6dfc" containerName="maas-api" Oct 30 15:28:33.811369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.810416 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacbbb8e-285b-4e9a-be21-ca29117f6dfc" containerName="maas-api" Oct 30 15:28:33.811369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.810621 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacbbb8e-285b-4e9a-be21-ca29117f6dfc" containerName="maas-api" Oct 30 15:28:33.816474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.816451 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.842567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.842541 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b236a6-openshift-default-dockercfg-gnhzx\"" Oct 30 15:28:33.843469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.843440 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz"] Oct 30 15:28:33.916468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.916403 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.916759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.916527 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.916759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.916555 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.917131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.916780 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.917131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.916963 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.917131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.916993 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.917440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.917174 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.917440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.917237 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.917440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.917311 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5g5z\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:33.932090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.930871 6554 patch_prober.go:29] interesting pod/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:28:33.932090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:33.930934 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:28:34.018294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018259 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018308 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5g5z\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018385 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018556 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018597 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018665 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018732 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018875 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018909 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.018960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.018923 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.019080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.019048 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.019200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.019178 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.019266 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.019223 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.019705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.019680 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.020727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.020704 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.020997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.020980 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.044004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.043973 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5g5z\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.052988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.052956 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token\") pod \"e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.148862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.148829 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:34.293224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.293188 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs"] Oct 30 15:28:34.298991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.298972 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.328713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.328687 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs"] Oct 30 15:28:34.343088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.343061 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-609709-openshift-default-dockercfg-78zbk\"" Oct 30 15:28:34.431848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.431789 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.431848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.431837 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.431911 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brws9\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.431981 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.432037 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.432079 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.432168 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.432206 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.432286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.432269 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.474681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.474654 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz"] Oct 30 15:28:34.474796 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:28:34.474700 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a97a21_862a_4156_9ffb_8dfadc5b306c.slice/crio-5bbaaefe0969c1778e28b32d148ac9e4ec55a8c6d9bf400794331e31bd8ff5d6 WatchSource:0}: Error finding container 5bbaaefe0969c1778e28b32d148ac9e4ec55a8c6d9bf400794331e31bd8ff5d6: Status 404 returned error can't find the container with id 5bbaaefe0969c1778e28b32d148ac9e4ec55a8c6d9bf400794331e31bd8ff5d6 Oct 30 15:28:34.477268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.477236 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:34.477325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.477299 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:34.477388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.477328 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:34.534499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534470 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.534626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534569 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.534698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534652 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.534756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534707 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.534823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534750 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brws9\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.534891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534829 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.534891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534864 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534917 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.534995 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.535007 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.535166 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.535340 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535571 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.535543 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.535866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.535841 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.536888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.536869 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.537603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.537575 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.560873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.560845 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brws9\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.561289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.561268 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token\") pod \"e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.613998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.613966 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:34.963947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.963915 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs"] Oct 30 15:28:34.964969 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:28:34.964940 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6bf27f_8644_461a_a224_dd833934e377.slice/crio-22d22fc1ae784e701a2c9001bc6ee0fcb4165d39aba57cc5d76565910b525166 WatchSource:0}: Error finding container 22d22fc1ae784e701a2c9001bc6ee0fcb4165d39aba57cc5d76565910b525166: Status 404 returned error can't find the container with id 22d22fc1ae784e701a2c9001bc6ee0fcb4165d39aba57cc5d76565910b525166 Oct 30 15:28:34.967783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.967747 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:34.967886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.967816 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:34.967886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:34.967843 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:35.380933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.380843 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" event={"ID":"1f6bf27f-8644-461a-a224-dd833934e377","Type":"ContainerStarted","Data":"39c4abd6cd6470e0e8e5b3fd342db87340a47b170eb251bfd3f49604b7911bb0"} Oct 30 15:28:35.380933 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.380899 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" event={"ID":"1f6bf27f-8644-461a-a224-dd833934e377","Type":"ContainerStarted","Data":"22d22fc1ae784e701a2c9001bc6ee0fcb4165d39aba57cc5d76565910b525166"} Oct 30 15:28:35.382872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.382846 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" event={"ID":"53a97a21-862a-4156-9ffb-8dfadc5b306c","Type":"ContainerStarted","Data":"646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2"} Oct 30 15:28:35.382999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.382894 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" event={"ID":"53a97a21-862a-4156-9ffb-8dfadc5b306c","Type":"ContainerStarted","Data":"5bbaaefe0969c1778e28b32d148ac9e4ec55a8c6d9bf400794331e31bd8ff5d6"} Oct 30 15:28:35.431606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.431553 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podStartSLOduration=1.431539419 podStartE2EDuration="1.431539419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:35.4283742 +0000 UTC m=+2419.387415606" watchObservedRunningTime="2026-10-30 15:28:35.431539419 +0000 UTC m=+2419.390580823" Oct 30 15:28:35.476970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.476921 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podStartSLOduration=2.476906844 podStartE2EDuration="2.476906844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:35.475078022 +0000 UTC m=+2419.434119430" watchObservedRunningTime="2026-10-30 15:28:35.476906844 +0000 UTC m=+2419.435948248" Oct 30 15:28:35.615045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:35.614824 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:36.180912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:36.180879 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:36.181324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:36.180974 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:36.390806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:36.390770 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:36.391827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:36.391808 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:28:36.616156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:36.616082 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 15:28:36.616156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:36.616133 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 15:28:37.615234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.615143 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 15:28:37.615234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.615220 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 15:28:37.688464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.688435 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:28:37.722686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.722657 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.722852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.722730 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.722852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.722761 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.722852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.722803 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.723024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.722867 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.723024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.722905 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:37.723150 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723030 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.723150 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723042 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:37.723150 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723113 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.723316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723197 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.723316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723251 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket\") pod \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\" (UID: \"2ff59fa2-d93a-451c-9db0-b6c7ae616001\") " Oct 30 15:28:37.723550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723510 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:28:37.723804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723754 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:37.723804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723792 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:37.723958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723914 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.723958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723935 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.723958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723949 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.724128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723962 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.724128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.723975 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.725698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.725672 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:37.725810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.725769 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "kube-api-access-v9zf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:28:37.725952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.725933 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:28:37.727245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.727215 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy" pod "2ff59fa2-d93a-451c-9db0-b6c7ae616001" (UID: "2ff59fa2-d93a-451c-9db0-b6c7ae616001"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:28:37.825724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.825689 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.825724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.825716 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.825724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.825725 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9zf8\" (UniqueName: \"kubernetes.io/projected/2ff59fa2-d93a-451c-9db0-b6c7ae616001-kube-api-access-v9zf8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:37.825957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:37.825734 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ff59fa2-d93a-451c-9db0-b6c7ae616001-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:28:38.406868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.406834 6554 generic.go:356] "Generic (PLEG): container finished" podID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerID="cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3" exitCode=0 Oct 30 15:28:38.407052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.406886 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" Oct 30 15:28:38.407052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.406920 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" event={"ID":"2ff59fa2-d93a-451c-9db0-b6c7ae616001","Type":"ContainerDied","Data":"cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3"} Oct 30 15:28:38.407052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.406980 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n" event={"ID":"2ff59fa2-d93a-451c-9db0-b6c7ae616001","Type":"ContainerDied","Data":"bff04b830e6f85d6ddae79b6e34c2f97473ab649ba9539b561167866b797f5bd"} Oct 30 15:28:38.407052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.407009 6554 scope.go:122] "RemoveContainer" containerID="cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3" Oct 30 15:28:38.417254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.417234 6554 scope.go:122] "RemoveContainer" containerID="cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3" Oct 30 15:28:38.417563 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:28:38.417540 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3\": container with ID starting with cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3 not found: ID does not exist" containerID="cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3" Oct 30 15:28:38.417660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.417571 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3"} err="failed to get container status \"cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3\": rpc error: code = NotFound desc = could not find container \"cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3\": container with ID starting with cedff76c1a35fa8f9cfdb0ceeac60eb6faa509c7eaf66dd820370abd4d1ed7f3 not found: ID does not exist" Oct 30 15:28:38.482478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.482406 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n"] Oct 30 15:28:38.495802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.495772 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-13009a-openshift-default-5645478d89-5tk7n"] Oct 30 15:28:38.614759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.614734 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 15:28:38.614883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:38.614787 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 15:28:39.206775 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:28:39.206742 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:28:39.614834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:39.614756 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 15:28:39.614834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:39.614812 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 15:28:40.294415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:40.282877 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" path="/var/lib/kubelet/pods/2ff59fa2-d93a-451c-9db0-b6c7ae616001/volumes" Oct 30 15:28:40.614279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:40.614196 6554 patch_prober.go:29] interesting pod/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" start-of-body= Oct 30 15:28:40.614279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:40.614254 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.190:15021/healthz/ready\": dial tcp 10.134.0.190:15021: connect: connection refused" Oct 30 15:28:41.714309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:41.714278 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:41.714815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:41.714721 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:41.715654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:41.715630 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:28:43.175611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.175582 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2"] Oct 30 15:28:43.176261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.176246 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" Oct 30 15:28:43.176310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.176263 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" Oct 30 15:28:43.176461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.176450 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff59fa2-d93a-451c-9db0-b6c7ae616001" containerName="istio-proxy" Oct 30 15:28:43.180053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.180037 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.199174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.199150 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-d57911-openshift-default-dockercfg-v99pm\"" Oct 30 15:28:43.217777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.217736 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2"] Oct 30 15:28:43.243681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.243650 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.243801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.243702 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.243922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.243906 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.243986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.243935 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.244150 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.244019 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z256c\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.244269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.244251 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.244435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.244409 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.244534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.244457 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.244534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.244505 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345401 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345432 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345458 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z256c\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345614 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345720 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345765 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345806 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345833 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345852 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.345895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345879 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.346067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.345992 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.346133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.346114 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.346239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.346209 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.346297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.346221 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.348205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.348183 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.348324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.348305 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.372400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.372272 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.372542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.372515 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z256c\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c\") pod \"e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.494786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.494716 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:43.818795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.818770 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2"] Oct 30 15:28:43.820004 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:28:43.819971 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0098b7_a192_4820_9aad_026cb459eb4a.slice/crio-f7b79dcafeb95719e40b2ae5606ae07b1aff51bfd0c0920a6e9de6640be61f33 WatchSource:0}: Error finding container f7b79dcafeb95719e40b2ae5606ae07b1aff51bfd0c0920a6e9de6640be61f33: Status 404 returned error can't find the container with id f7b79dcafeb95719e40b2ae5606ae07b1aff51bfd0c0920a6e9de6640be61f33 Oct 30 15:28:43.822686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.822654 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:43.822758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.822725 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:43.822758 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:43.822751 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:44.450048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:44.450015 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" event={"ID":"7e0098b7-a192-4820-9aad-026cb459eb4a","Type":"ContainerStarted","Data":"ef1b495ef88c1815d41a016573f129e177b144aa9393496f4bd24a503dd32f02"} Oct 30 15:28:44.450540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:44.450073 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" event={"ID":"7e0098b7-a192-4820-9aad-026cb459eb4a","Type":"ContainerStarted","Data":"f7b79dcafeb95719e40b2ae5606ae07b1aff51bfd0c0920a6e9de6640be61f33"} Oct 30 15:28:44.496273 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:44.496248 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:44.497893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:44.497859 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 15:28:44.498029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:44.497911 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 15:28:44.513713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:44.513672 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podStartSLOduration=1.5136593189999998 podStartE2EDuration="1.513659319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:28:44.511140866 +0000 UTC m=+2428.470182285" watchObservedRunningTime="2026-10-30 15:28:44.513659319 +0000 UTC m=+2428.472700724" Oct 30 15:28:45.495526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:45.495486 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 15:28:45.496033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:45.495562 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 15:28:46.495872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:46.495838 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 15:28:46.496308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:46.495905 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 15:28:47.514536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:47.511248 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 15:28:47.514536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:47.511308 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 15:28:48.495329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:48.495296 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 15:28:48.495535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:48.495389 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 15:28:49.496573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:49.496325 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 15:28:49.496573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:49.496416 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 15:28:50.773372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:50.766562 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:50.773372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:50.767814 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:50.774210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:50.774186 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:28:58.955657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:58.955554 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd"] Oct 30 15:28:58.962575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:58.962547 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:58.993525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:58.985638 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd"] Oct 30 15:28:58.993525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:58.992747 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf437b-openshift-default-dockercfg-cg4bz\"" Oct 30 15:28:59.067970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.067934 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.067991 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068076 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068121 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068270 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068291 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068317 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n5nm\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068334 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.068576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.068550 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170317 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170386 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170430 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n5nm\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170455 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170569 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170595 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170646 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170708 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170781 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.170847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.170825 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.171132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.171043 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.171401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.171338 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.171666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.171645 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.171771 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.171752 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.173874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.173843 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.174209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.174191 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.195815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.195791 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.196003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.195984 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n5nm\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm\") pod \"e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.295947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.295880 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:28:59.622649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.622619 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd"] Oct 30 15:28:59.622802 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:28:59.622724 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2ec6b9_f6e2_418c_baee_f6dcd25e2889.slice/crio-d61fcb445f0924ad6dcb097949a1d6aafd6764fd484ddac48285108bda83288e WatchSource:0}: Error finding container d61fcb445f0924ad6dcb097949a1d6aafd6764fd484ddac48285108bda83288e: Status 404 returned error can't find the container with id d61fcb445f0924ad6dcb097949a1d6aafd6764fd484ddac48285108bda83288e Oct 30 15:28:59.625343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.625309 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:59.625460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.625389 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:28:59.625460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:28:59.625421 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:00.635009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:00.634973 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" event={"ID":"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889","Type":"ContainerStarted","Data":"7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df"} Oct 30 15:29:00.635433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:00.635029 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" event={"ID":"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889","Type":"ContainerStarted","Data":"d61fcb445f0924ad6dcb097949a1d6aafd6764fd484ddac48285108bda83288e"} Oct 30 15:29:00.703000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:00.702955 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podStartSLOduration=2.702941928 podStartE2EDuration="2.702941928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:00.700211299 +0000 UTC m=+2444.659252706" watchObservedRunningTime="2026-10-30 15:29:00.702941928 +0000 UTC m=+2444.661983332" Oct 30 15:29:01.296186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:01.296154 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:29:01.301655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:01.301631 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:29:01.641697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:01.641670 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:29:01.642681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:01.642661 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:29:05.342124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.342082 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr"] Oct 30 15:29:05.347111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.347087 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.402324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.402296 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf437b\"" Oct 30 15:29:05.402532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.402510 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf437b-dockercfg-lpgqw\"" Oct 30 15:29:05.406418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.406394 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr"] Oct 30 15:29:05.417711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.417686 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.418080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.418057 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxmtg\" (UniqueName: \"kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.528679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.528628 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.529944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.529906 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxmtg\" (UniqueName: \"kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.531669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.529786 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.586366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.582202 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxmtg\" (UniqueName: \"kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg\") pod \"payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.667770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.667738 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:05.977257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:05.977226 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr"] Oct 30 15:29:05.978266 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:29:05.978243 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51ab9aa_7289_40d4_9a6d_8b1d0cb39224.slice/crio-d94c74879cd8d361e0004475d6c1e556fec096e3a5e0ea8594ac43f711a93242 WatchSource:0}: Error finding container d94c74879cd8d361e0004475d6c1e556fec096e3a5e0ea8594ac43f711a93242: Status 404 returned error can't find the container with id d94c74879cd8d361e0004475d6c1e556fec096e3a5e0ea8594ac43f711a93242 Oct 30 15:29:06.676541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:06.676498 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" event={"ID":"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224","Type":"ContainerStarted","Data":"2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb"} Oct 30 15:29:06.676930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:06.676563 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" event={"ID":"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224","Type":"ContainerStarted","Data":"d94c74879cd8d361e0004475d6c1e556fec096e3a5e0ea8594ac43f711a93242"} Oct 30 15:29:06.676930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:06.676666 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:06.722722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:06.722675 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" podStartSLOduration=1.722662566 podStartE2EDuration="1.722662566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:06.719063417 +0000 UTC m=+2450.678104823" watchObservedRunningTime="2026-10-30 15:29:06.722662566 +0000 UTC m=+2450.681703971" Oct 30 15:29:09.202541 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:29:09.202495 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:29:17.687073 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:17.687040 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:29:18.561613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.561568 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q"] Oct 30 15:29:18.599123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.599086 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q"] Oct 30 15:29:18.599422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.599221 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767072 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnpfs\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767106 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767128 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767148 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767282 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767431 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767456 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767583 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.767690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.767615 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869334 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869384 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869441 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869457 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869612 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnpfs\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869642 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869668 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869695 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.869820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869790 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.870058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.869920 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.870120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.870059 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.870120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.870070 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.870232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.870187 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.870334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.870311 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.871701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.871683 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.872179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.872156 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.894177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.894147 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.894419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.894399 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnpfs\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs\") pod \"e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:18.926947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:18.926926 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:19.250068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.250036 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q"] Oct 30 15:29:19.250891 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:29:19.250866 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb381190_c5f6_46c1_a14e_51a4c413aaae.slice/crio-7aac56259f9a873e73bc3d0dedb4014ac60e5178cc6e1cc933044c76b3e017f0 WatchSource:0}: Error finding container 7aac56259f9a873e73bc3d0dedb4014ac60e5178cc6e1cc933044c76b3e017f0: Status 404 returned error can't find the container with id 7aac56259f9a873e73bc3d0dedb4014ac60e5178cc6e1cc933044c76b3e017f0 Oct 30 15:29:19.263754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.263711 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:19.263876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.263800 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:19.263876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.263845 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:19.853379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.849855 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" event={"ID":"db381190-c5f6-46c1-a14e-51a4c413aaae","Type":"ContainerStarted","Data":"e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5"} Oct 30 15:29:19.853379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.849959 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" event={"ID":"db381190-c5f6-46c1-a14e-51a4c413aaae","Type":"ContainerStarted","Data":"7aac56259f9a873e73bc3d0dedb4014ac60e5178cc6e1cc933044c76b3e017f0"} Oct 30 15:29:19.901558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.901498 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podStartSLOduration=1.901478806 podStartE2EDuration="1.901478806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:19.899541023 +0000 UTC m=+2463.858582429" watchObservedRunningTime="2026-10-30 15:29:19.901478806 +0000 UTC m=+2463.860520213" Oct 30 15:29:19.927519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:19.927485 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:20.928607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:20.928569 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 15:29:20.929008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:20.928638 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 15:29:21.927427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:21.927391 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 15:29:21.927624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:21.927451 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 15:29:22.927939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:22.927910 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 15:29:22.928397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:22.927959 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 15:29:23.927360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:23.927312 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 15:29:23.927536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:23.927380 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 15:29:24.927913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:24.927878 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 15:29:24.928305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:24.927941 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 15:29:25.970639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:25.970595 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:25.971156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:25.970958 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:25.972284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:25.972261 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:29:35.394080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:35.394046 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq"] Oct 30 15:29:35.428759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:35.428730 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5"] Oct 30 15:29:39.202703 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:29:39.202664 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:29:40.394851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.394811 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" podUID="e410084b-3ff4-4cb5-b190-50b5682837bd" containerName="payload-processing" containerID="cri-o://15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0" gracePeriod=25 Oct 30 15:29:40.429779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.429750 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" podUID="3d8e017f-aed4-4f08-986f-433c5fdfc77d" containerName="payload-pre-processing" containerID="cri-o://41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a" gracePeriod=25 Oct 30 15:29:40.703951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.703920 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:29:40.751559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.751524 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:29:40.762899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.762874 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume\") pod \"e410084b-3ff4-4cb5-b190-50b5682837bd\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " Oct 30 15:29:40.763040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.762963 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254\" (UniqueName: \"kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254\") pod \"e410084b-3ff4-4cb5-b190-50b5682837bd\" (UID: \"e410084b-3ff4-4cb5-b190-50b5682837bd\") " Oct 30 15:29:40.763283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.763257 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume" pod "e410084b-3ff4-4cb5-b190-50b5682837bd" (UID: "e410084b-3ff4-4cb5-b190-50b5682837bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:29:40.763667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.763648 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e410084b-3ff4-4cb5-b190-50b5682837bd-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:29:40.765016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.764993 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254" pod "e410084b-3ff4-4cb5-b190-50b5682837bd" (UID: "e410084b-3ff4-4cb5-b190-50b5682837bd"). InnerVolumeSpecName "kube-api-access-z5254". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:29:40.864547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.864518 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume\") pod \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " Oct 30 15:29:40.864690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.864663 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996\" (UniqueName: \"kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996\") pod \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\" (UID: \"3d8e017f-aed4-4f08-986f-433c5fdfc77d\") " Oct 30 15:29:40.864881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.864858 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume" pod "3d8e017f-aed4-4f08-986f-433c5fdfc77d" (UID: "3d8e017f-aed4-4f08-986f-433c5fdfc77d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:29:40.865268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.865248 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d8e017f-aed4-4f08-986f-433c5fdfc77d-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:29:40.865378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.865272 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5254\" (UniqueName: \"kubernetes.io/projected/e410084b-3ff4-4cb5-b190-50b5682837bd-kube-api-access-z5254\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:29:40.866788 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.866738 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996" pod "3d8e017f-aed4-4f08-986f-433c5fdfc77d" (UID: "3d8e017f-aed4-4f08-986f-433c5fdfc77d"). InnerVolumeSpecName "kube-api-access-cm996". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:29:40.966317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:40.966262 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm996\" (UniqueName: \"kubernetes.io/projected/3d8e017f-aed4-4f08-986f-433c5fdfc77d-kube-api-access-cm996\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:29:41.032398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.032370 6554 generic.go:356] "Generic (PLEG): container finished" podID="e410084b-3ff4-4cb5-b190-50b5682837bd" containerID="15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0" exitCode=0 Oct 30 15:29:41.032496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.032411 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" Oct 30 15:29:41.032496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.032438 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" event={"ID":"e410084b-3ff4-4cb5-b190-50b5682837bd","Type":"ContainerDied","Data":"15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0"} Oct 30 15:29:41.032496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.032491 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq" event={"ID":"e410084b-3ff4-4cb5-b190-50b5682837bd","Type":"ContainerDied","Data":"b89d9feab9366fb420171e5fb3f26eefc7a2b76cce0abb17b49e9675f2119abd"} Oct 30 15:29:41.032643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.032514 6554 scope.go:122] "RemoveContainer" containerID="15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0" Oct 30 15:29:41.033958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.033935 6554 generic.go:356] "Generic (PLEG): container finished" podID="3d8e017f-aed4-4f08-986f-433c5fdfc77d" containerID="41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a" exitCode=0 Oct 30 15:29:41.033958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.033960 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" event={"ID":"3d8e017f-aed4-4f08-986f-433c5fdfc77d","Type":"ContainerDied","Data":"41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a"} Oct 30 15:29:41.034198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.033986 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" Oct 30 15:29:41.034198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.034007 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5" event={"ID":"3d8e017f-aed4-4f08-986f-433c5fdfc77d","Type":"ContainerDied","Data":"be4964d01c1af0ee0b46e820d6a2a5869b89f5a97cdc6fb5bb26c92b79bcfc51"} Oct 30 15:29:41.043563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.043432 6554 scope.go:122] "RemoveContainer" containerID="15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0" Oct 30 15:29:41.043784 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:29:41.043756 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0\": container with ID starting with 15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0 not found: ID does not exist" containerID="15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0" Oct 30 15:29:41.043879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.043792 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0"} err="failed to get container status \"15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0\": rpc error: code = NotFound desc = could not find container \"15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0\": container with ID starting with 15c0979187a266314c83ba8a6b273a6d44c737aa85feec1c1f3feeb76cc173c0 not found: ID does not exist" Oct 30 15:29:41.043879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.043815 6554 scope.go:122] "RemoveContainer" containerID="41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a" Oct 30 15:29:41.053976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.053952 6554 scope.go:122] "RemoveContainer" containerID="41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a" Oct 30 15:29:41.054228 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:29:41.054210 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a\": container with ID starting with 41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a not found: ID does not exist" containerID="41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a" Oct 30 15:29:41.054289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.054234 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a"} err="failed to get container status \"41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a\": rpc error: code = NotFound desc = could not find container \"41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a\": container with ID starting with 41d8f18abb17cdfe8c48971ad96b2f7aafb771c547f08952acaa4e0c833e1d0a not found: ID does not exist" Oct 30 15:29:41.090096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.090074 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq"] Oct 30 15:29:41.094196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.094174 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7aede6-5c98588fd8-drpwq"] Oct 30 15:29:41.131596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.131567 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5"] Oct 30 15:29:41.135609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:41.135587 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7aede6-688fb8c79f-lq7v5"] Oct 30 15:29:42.155532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.155500 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8e017f-aed4-4f08-986f-433c5fdfc77d" path="/var/lib/kubelet/pods/3d8e017f-aed4-4f08-986f-433c5fdfc77d/volumes" Oct 30 15:29:42.155939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.155924 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e410084b-3ff4-4cb5-b190-50b5682837bd" path="/var/lib/kubelet/pods/e410084b-3ff4-4cb5-b190-50b5682837bd/volumes" Oct 30 15:29:42.821480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.821455 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n"] Oct 30 15:29:42.822161 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.822146 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3d8e017f-aed4-4f08-986f-433c5fdfc77d" containerName="payload-pre-processing" Oct 30 15:29:42.822224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.822163 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8e017f-aed4-4f08-986f-433c5fdfc77d" containerName="payload-pre-processing" Oct 30 15:29:42.822224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.822176 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e410084b-3ff4-4cb5-b190-50b5682837bd" containerName="payload-processing" Oct 30 15:29:42.822224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.822181 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e410084b-3ff4-4cb5-b190-50b5682837bd" containerName="payload-processing" Oct 30 15:29:42.822324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.822278 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8e017f-aed4-4f08-986f-433c5fdfc77d" containerName="payload-pre-processing" Oct 30 15:29:42.822324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.822288 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e410084b-3ff4-4cb5-b190-50b5682837bd" containerName="payload-processing" Oct 30 15:29:42.825899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.825883 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.860996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.860958 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f3a578-serving-cert\"" Oct 30 15:29:42.864541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.864518 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n"] Oct 30 15:29:42.890934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.890910 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.891053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.890970 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.891118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.891090 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.891250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.891235 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtwlc\" (UniqueName: \"kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.992620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.992593 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtwlc\" (UniqueName: \"kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.992742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.992680 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.992742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.992727 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.992857 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:29:42.992810 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-serving-cert: secret "maas-api-e2e-ipp-a-f3a578-serving-cert" not found Oct 30 15:29:42.992927 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:29:42.992864 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls podName:dcf9f4c3-828f-4c14-8d2f-690f8f100e6d nodeName:}" failed. No retries permitted until 2026-10-30 15:29:43.49284881 +0000 UTC m=+2487.451890216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls") pod "maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" (UID: "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d") : secret "maas-api-e2e-ipp-a-f3a578-serving-cert" not found Oct 30 15:29:42.992927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.992855 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.993298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.993276 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:42.995220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:42.995200 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:43.017421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:43.017395 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtwlc\" (UniqueName: \"kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:43.499843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:43.499812 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:43.502257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:43.502235 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:43.748550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:43.748512 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:44.061675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:44.061645 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n"] Oct 30 15:29:44.062525 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:29:44.062497 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf9f4c3_828f_4c14_8d2f_690f8f100e6d.slice/crio-588c993a4a277c3bb22f671fa366a4dbd529fce737c275fa6278aeff164e41d5 WatchSource:0}: Error finding container 588c993a4a277c3bb22f671fa366a4dbd529fce737c275fa6278aeff164e41d5: Status 404 returned error can't find the container with id 588c993a4a277c3bb22f671fa366a4dbd529fce737c275fa6278aeff164e41d5 Oct 30 15:29:45.064413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:45.064378 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" event={"ID":"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d","Type":"ContainerStarted","Data":"469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba"} Oct 30 15:29:45.064890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:45.064441 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" event={"ID":"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d","Type":"ContainerStarted","Data":"588c993a4a277c3bb22f671fa366a4dbd529fce737c275fa6278aeff164e41d5"} Oct 30 15:29:45.064890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:45.064470 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:45.113223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:45.113177 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" podStartSLOduration=2.729524894 podStartE2EDuration="3.113165172s" totalImagesPullingTime="383.640278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:45.109336426 +0000 UTC m=+2489.068377831" watchObservedRunningTime="2026-10-30 15:29:45.113165172 +0000 UTC m=+2489.072206577" Oct 30 15:29:46.094959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.094929 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k"] Oct 30 15:29:46.099646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.099624 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.120768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.120734 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-09e8eb-openshift-default-dockercfg-9727f\"" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.153990 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154052 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154081 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154127 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154162 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154260 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154411 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154455 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptf5\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.156380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.154551 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.183921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.183888 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k"] Oct 30 15:29:46.256504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256465 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.256504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256509 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.256747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256568 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.256747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256647 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.256859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256786 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptf5\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.256939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256919 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.256964 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257069 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257105 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257128 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257295 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257449 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257531 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.257962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.257940 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.259031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.259003 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.259121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.259091 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.291365 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.291328 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.293008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.292978 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptf5\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5\") pod \"e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.413606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.413580 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:46.743212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.743181 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k"] Oct 30 15:29:46.743877 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:29:46.743845 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4909339b_d70f_4224_9b44_3dbcefd347ca.slice/crio-bb3068e123f68e67fb2cab0fe8aaec114141509586df8dc06eee028f097ec050 WatchSource:0}: Error finding container bb3068e123f68e67fb2cab0fe8aaec114141509586df8dc06eee028f097ec050: Status 404 returned error can't find the container with id bb3068e123f68e67fb2cab0fe8aaec114141509586df8dc06eee028f097ec050 Oct 30 15:29:46.746842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.746809 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:46.746915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.746873 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:46.746915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:46.746902 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:29:47.079561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:47.079474 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" event={"ID":"4909339b-d70f-4224-9b44-3dbcefd347ca","Type":"ContainerStarted","Data":"fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5"} Oct 30 15:29:47.079561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:47.079534 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" event={"ID":"4909339b-d70f-4224-9b44-3dbcefd347ca","Type":"ContainerStarted","Data":"bb3068e123f68e67fb2cab0fe8aaec114141509586df8dc06eee028f097ec050"} Oct 30 15:29:47.135911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:47.135860 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podStartSLOduration=1.135845643 podStartE2EDuration="1.135845643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:29:47.133334496 +0000 UTC m=+2491.092375923" watchObservedRunningTime="2026-10-30 15:29:47.135845643 +0000 UTC m=+2491.094887047" Oct 30 15:29:47.414700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:47.414670 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:47.416567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:47.416541 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 15:29:47.416735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:47.416588 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 15:29:48.415008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:48.414979 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 15:29:48.415446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:48.415030 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 15:29:49.414236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:49.414192 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 15:29:49.414472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:49.414251 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 15:29:50.414085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:50.414046 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 15:29:50.414601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:50.414095 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 15:29:51.083709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:51.083678 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:29:51.414473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:51.414441 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 15:29:51.414880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:51.414504 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 15:29:52.414755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:52.414722 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 15:29:52.415170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:52.414769 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 15:29:53.437103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:53.437071 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:53.438039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:53.438015 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:29:53.440539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:29:53.440518 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:30:00.212608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.212570 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt"] Oct 30 15:30:00.230681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.230641 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns"] Oct 30 15:30:00.233939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.233909 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.236938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.236806 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.243638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.243616 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft"] Oct 30 15:30:00.251752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.251727 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns"] Oct 30 15:30:00.251888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.251810 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.262255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.261608 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c"] Oct 30 15:30:00.269228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.269203 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt"] Oct 30 15:30:00.269380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.269248 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf"] Oct 30 15:30:00.274973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.271230 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.276898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.276874 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.281428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.281405 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft"] Oct 30 15:30:00.287423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.287403 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c"] Oct 30 15:30:00.291106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.291072 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrs8\" (UniqueName: \"kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.291668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.291645 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ch7f\" (UniqueName: \"kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.292088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.292067 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.292341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.292323 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.292550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.292533 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qg8g\" (UniqueName: \"kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.292712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.292697 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.292945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.292931 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj"] Oct 30 15:30:00.296696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.296676 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:30:00.327569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.327539 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl"] Oct 30 15:30:00.330635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.330614 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.337244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.337223 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf"] Oct 30 15:30:00.337442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.337427 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4"] Oct 30 15:30:00.339012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.338986 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.364457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.364430 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt"] Oct 30 15:30:00.365687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.365666 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.377838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.377716 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.378707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.377871 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl"] Oct 30 15:30:00.378707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.377913 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj"] Oct 30 15:30:00.378707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.377940 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt"] Oct 30 15:30:00.384898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.383333 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4"] Oct 30 15:30:00.394920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.394891 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.396419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.396395 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.396602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.396581 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.397443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397423 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qg8g\" (UniqueName: \"kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.397614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397330 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.397685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397645 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.397749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397731 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr6q2\" (UniqueName: \"kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2\") pod \"maas-api-key-cleanup-29889570-rxvnj\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.397810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397763 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrs8\" (UniqueName: \"kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.397810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397798 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bzcc\" (UniqueName: \"kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.397934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397829 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889570-rxvnj\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.397934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397869 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ch7f\" (UniqueName: \"kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.397934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.397896 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.398465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.398435 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcv8k\" (UniqueName: \"kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.398568 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.398499 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.398568 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.398537 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrkx9\" (UniqueName: \"kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9\") pod \"maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.398701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.398573 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.398701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.398613 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhkd\" (UniqueName: \"kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.398876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.398854 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.399508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.399485 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.427338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.427313 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ch7f\" (UniqueName: \"kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f\") pod \"maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.428952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.428927 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qg8g\" (UniqueName: \"kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g\") pod \"maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.430121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.430098 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrs8\" (UniqueName: \"kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.503048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.502957 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr6q2\" (UniqueName: \"kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2\") pod \"maas-api-key-cleanup-29889570-rxvnj\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.503697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.503601 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bzcc\" (UniqueName: \"kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.504957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.504055 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889570-rxvnj\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.505157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.505138 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.506219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.506199 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcv8k\" (UniqueName: \"kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.506670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.506649 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.507524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.507504 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrkx9\" (UniqueName: \"kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9\") pod \"maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.507926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.507906 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhkd\" (UniqueName: \"kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.508398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.508342 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.510277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.510258 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.510489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.510470 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwl58\" (UniqueName: \"kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.510675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.510659 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.511844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.504918 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889570-rxvnj\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.512031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.505972 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.512165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.507439 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.512289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.510197 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.512434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.511409 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.527975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.527945 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr6q2\" (UniqueName: \"kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2\") pod \"maas-api-key-cleanup-29889570-rxvnj\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.528199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.528175 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bzcc\" (UniqueName: \"kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.530724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.530696 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcv8k\" (UniqueName: \"kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k\") pod \"maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.531745 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.531701 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrkx9\" (UniqueName: \"kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9\") pod \"maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.532308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.532288 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhkd\" (UniqueName: \"kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd\") pod \"maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.601336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.601302 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:00.612447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.612417 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.612599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.612459 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwl58\" (UniqueName: \"kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.613230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.613201 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.614973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.614949 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:00.637557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.637529 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwl58\" (UniqueName: \"kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:00.637796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.637775 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:00.650846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.650657 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:00.695804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.692754 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:00.695804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.693219 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:00.703809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.696181 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:00.706486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.706459 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:00.715511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:00.715275 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:01.188419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.188015 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt"] Oct 30 15:30:01.219507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.219455 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft"] Oct 30 15:30:01.294404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.294313 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerStarted","Data":"387ef1899f5c23a9bf9aec7a921c694274cdc954ca51db02666253e94faed941"} Oct 30 15:30:01.402104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.402077 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns"] Oct 30 15:30:01.406621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.406594 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c"] Oct 30 15:30:01.415712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.415522 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt"] Oct 30 15:30:01.440017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.439993 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf"] Oct 30 15:30:01.491641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.491601 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl"] Oct 30 15:30:01.496412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.496386 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj"] Oct 30 15:30:01.497729 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:30:01.497701 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66217a85_3544_4350_8c0b_a4873f0b2962.slice/crio-72d8ae855e20be0d9662f0612bc9460648e2eed90f93ecee67a14553b574a693 WatchSource:0}: Error finding container 72d8ae855e20be0d9662f0612bc9460648e2eed90f93ecee67a14553b574a693: Status 404 returned error can't find the container with id 72d8ae855e20be0d9662f0612bc9460648e2eed90f93ecee67a14553b574a693 Oct 30 15:30:01.498098 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:30:01.498072 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1cd589_4fe8_4a98_a633_0129d9d4bbb1.slice/crio-8ef1a08ea520769ad78b61fc2af4a1717b4f58599d80de1854fe8b2470782c1b WatchSource:0}: Error finding container 8ef1a08ea520769ad78b61fc2af4a1717b4f58599d80de1854fe8b2470782c1b: Status 404 returned error can't find the container with id 8ef1a08ea520769ad78b61fc2af4a1717b4f58599d80de1854fe8b2470782c1b Oct 30 15:30:01.500599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:01.500509 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4"] Oct 30 15:30:02.304424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.304389 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerStarted","Data":"553f7ffa4df05c8f38688dfed4624a881a3d08fe82038c8469f1d0c078461d15"} Oct 30 15:30:02.304796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.304450 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerStarted","Data":"8ef1a08ea520769ad78b61fc2af4a1717b4f58599d80de1854fe8b2470782c1b"} Oct 30 15:30:02.306297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.306261 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerStarted","Data":"1f2e834271aa8fbe2097789dd20078680d31469655fe103582756ae5eacba2bd"} Oct 30 15:30:02.306423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.306310 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerStarted","Data":"47bfbcc4114fb31c22bab578f4bbd33f5b6815e1c05f12fb68fb1a82ea59f187"} Oct 30 15:30:02.307709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.307686 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerStarted","Data":"0680f14a7a603e0b09abe7c975a6d7f8cf5dfef8d7785ecbab8faf155626158f"} Oct 30 15:30:02.309262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.309237 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerStarted","Data":"95e8525f21d36aff7732dd45f1ad5ea5b9bd4b07081557fff310021cfdfe62a9"} Oct 30 15:30:02.309367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.309279 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerStarted","Data":"53417d6977fdf8806aedb8ce2c3aad7ca0f8a06f6916f9d12dc3ac1b23bb10cf"} Oct 30 15:30:02.310765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.310743 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerStarted","Data":"772763d83934f48c41bea615deaf8b4aba0f9e0468288650363402e0d6527a6a"} Oct 30 15:30:02.310818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.310778 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerStarted","Data":"2dab19bed57903e4ab936f28230419d95565f82dde659ebe3b76d5378e2bba46"} Oct 30 15:30:02.312314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.312295 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerStarted","Data":"6b4547a4cf375a1322ea268d6136978707517d67c274ab1f954a49d1cb24fcf9"} Oct 30 15:30:02.312381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.312331 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerStarted","Data":"f5665c0cb2dee4db59bcd0a64dce31c35e4ce351392313290ecd36bd1b82a1d1"} Oct 30 15:30:02.313855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.313825 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerStarted","Data":"dbff1c72f2a9b8a3e1d81dd6edd8741eb701fbc4584ce01298a8a1a4dcad78d3"} Oct 30 15:30:02.313948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.313872 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerStarted","Data":"0e08f89d40bdf0ba5677a244f22845633d164c7b5db1c0810e6598fee785bf43"} Oct 30 15:30:02.315424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.315404 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerStarted","Data":"f68716082ed48c90fb1a4c6637855f36daada28202decbe03962ea1248f3238d"} Oct 30 15:30:02.315502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.315437 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerStarted","Data":"72d8ae855e20be0d9662f0612bc9460648e2eed90f93ecee67a14553b574a693"} Oct 30 15:30:02.316896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.316872 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerStarted","Data":"1870a5b430c257a5f6ed1b0883dd77a10d93b6781c4c6599d7c33ab54405260d"} Oct 30 15:30:02.316980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.316916 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerStarted","Data":"e59e4b80ea641a7bdf742264215a0914fe626a0655209a3d4b6d2b9b50d309d8"} Oct 30 15:30:02.352040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.351995 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" podStartSLOduration=2.351982515 podStartE2EDuration="2.351982515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.349955428 +0000 UTC m=+2506.308996833" watchObservedRunningTime="2026-10-30 15:30:02.351982515 +0000 UTC m=+2506.311023919" Oct 30 15:30:02.483583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.483529 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" podStartSLOduration=2.483515123 podStartE2EDuration="2.483515123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.412932884 +0000 UTC m=+2506.371974290" watchObservedRunningTime="2026-10-30 15:30:02.483515123 +0000 UTC m=+2506.442556528" Oct 30 15:30:02.485760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.485341 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" podStartSLOduration=2.4853285 podStartE2EDuration="2.4853285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.471486116 +0000 UTC m=+2506.430527525" watchObservedRunningTime="2026-10-30 15:30:02.4853285 +0000 UTC m=+2506.444369909" Oct 30 15:30:02.520387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.520328 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" podStartSLOduration=2.520314345 podStartE2EDuration="2.520314345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.517120651 +0000 UTC m=+2506.476162057" watchObservedRunningTime="2026-10-30 15:30:02.520314345 +0000 UTC m=+2506.479355750" Oct 30 15:30:02.587392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.587245 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" podStartSLOduration=2.587226149 podStartE2EDuration="2.587226149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.564632766 +0000 UTC m=+2506.523674173" watchObservedRunningTime="2026-10-30 15:30:02.587226149 +0000 UTC m=+2506.546267557" Oct 30 15:30:02.612914 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.612864 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" podStartSLOduration=2.6128348 podStartE2EDuration="2.6128348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.611075577 +0000 UTC m=+2506.570116984" watchObservedRunningTime="2026-10-30 15:30:02.6128348 +0000 UTC m=+2506.571876206" Oct 30 15:30:02.687470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.687419 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" podStartSLOduration=2.687405766 podStartE2EDuration="2.687405766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.665399549 +0000 UTC m=+2506.624440955" watchObservedRunningTime="2026-10-30 15:30:02.687405766 +0000 UTC m=+2506.646447171" Oct 30 15:30:02.714577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.714515 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" podStartSLOduration=2.7145001300000002 podStartE2EDuration="2.71450013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.711797604 +0000 UTC m=+2506.670839020" watchObservedRunningTime="2026-10-30 15:30:02.71450013 +0000 UTC m=+2506.673541536" Oct 30 15:30:02.787605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:02.787546 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" podStartSLOduration=2.787527346 podStartE2EDuration="2.787527346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:02.78501013 +0000 UTC m=+2506.744051536" watchObservedRunningTime="2026-10-30 15:30:02.787527346 +0000 UTC m=+2506.746568755" Oct 30 15:30:03.351228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.351009 6554 generic.go:356] "Generic (PLEG): container finished" podID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerID="553f7ffa4df05c8f38688dfed4624a881a3d08fe82038c8469f1d0c078461d15" exitCode=60 Oct 30 15:30:03.351228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.351128 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerDied","Data":"553f7ffa4df05c8f38688dfed4624a881a3d08fe82038c8469f1d0c078461d15"} Oct 30 15:30:03.351746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.351701 6554 scope.go:122] "RemoveContainer" containerID="553f7ffa4df05c8f38688dfed4624a881a3d08fe82038c8469f1d0c078461d15" Oct 30 15:30:03.383063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.381815 6554 generic.go:356] "Generic (PLEG): container finished" podID="8a856bbd-fcbd-4205-be12-58918509137d" containerID="1f2e834271aa8fbe2097789dd20078680d31469655fe103582756ae5eacba2bd" exitCode=60 Oct 30 15:30:03.383063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.381976 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerDied","Data":"1f2e834271aa8fbe2097789dd20078680d31469655fe103582756ae5eacba2bd"} Oct 30 15:30:03.383063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.382553 6554 scope.go:122] "RemoveContainer" containerID="1f2e834271aa8fbe2097789dd20078680d31469655fe103582756ae5eacba2bd" Oct 30 15:30:03.390217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.390191 6554 generic.go:356] "Generic (PLEG): container finished" podID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerID="0680f14a7a603e0b09abe7c975a6d7f8cf5dfef8d7785ecbab8faf155626158f" exitCode=60 Oct 30 15:30:03.390333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.390321 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerDied","Data":"0680f14a7a603e0b09abe7c975a6d7f8cf5dfef8d7785ecbab8faf155626158f"} Oct 30 15:30:03.390775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.390758 6554 scope.go:122] "RemoveContainer" containerID="0680f14a7a603e0b09abe7c975a6d7f8cf5dfef8d7785ecbab8faf155626158f" Oct 30 15:30:03.393306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.393273 6554 generic.go:356] "Generic (PLEG): container finished" podID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerID="95e8525f21d36aff7732dd45f1ad5ea5b9bd4b07081557fff310021cfdfe62a9" exitCode=60 Oct 30 15:30:03.393935 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.393390 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerDied","Data":"95e8525f21d36aff7732dd45f1ad5ea5b9bd4b07081557fff310021cfdfe62a9"} Oct 30 15:30:03.394403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.393550 6554 scope.go:122] "RemoveContainer" containerID="95e8525f21d36aff7732dd45f1ad5ea5b9bd4b07081557fff310021cfdfe62a9" Oct 30 15:30:03.395525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.395505 6554 generic.go:356] "Generic (PLEG): container finished" podID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerID="772763d83934f48c41bea615deaf8b4aba0f9e0468288650363402e0d6527a6a" exitCode=60 Oct 30 15:30:03.395602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.395591 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerDied","Data":"772763d83934f48c41bea615deaf8b4aba0f9e0468288650363402e0d6527a6a"} Oct 30 15:30:03.395944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.395928 6554 scope.go:122] "RemoveContainer" containerID="772763d83934f48c41bea615deaf8b4aba0f9e0468288650363402e0d6527a6a" Oct 30 15:30:03.398641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.398623 6554 generic.go:356] "Generic (PLEG): container finished" podID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerID="6b4547a4cf375a1322ea268d6136978707517d67c274ab1f954a49d1cb24fcf9" exitCode=60 Oct 30 15:30:03.399337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.398785 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerDied","Data":"6b4547a4cf375a1322ea268d6136978707517d67c274ab1f954a49d1cb24fcf9"} Oct 30 15:30:03.399337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.399166 6554 scope.go:122] "RemoveContainer" containerID="6b4547a4cf375a1322ea268d6136978707517d67c274ab1f954a49d1cb24fcf9" Oct 30 15:30:03.406220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.406134 6554 generic.go:356] "Generic (PLEG): container finished" podID="261ecca1-0870-48a2-b233-5364600435b4" containerID="dbff1c72f2a9b8a3e1d81dd6edd8741eb701fbc4584ce01298a8a1a4dcad78d3" exitCode=60 Oct 30 15:30:03.406220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.406208 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerDied","Data":"dbff1c72f2a9b8a3e1d81dd6edd8741eb701fbc4584ce01298a8a1a4dcad78d3"} Oct 30 15:30:03.406608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.406591 6554 scope.go:122] "RemoveContainer" containerID="dbff1c72f2a9b8a3e1d81dd6edd8741eb701fbc4584ce01298a8a1a4dcad78d3" Oct 30 15:30:03.408428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.408407 6554 generic.go:356] "Generic (PLEG): container finished" podID="66217a85-3544-4350-8c0b-a4873f0b2962" containerID="f68716082ed48c90fb1a4c6637855f36daada28202decbe03962ea1248f3238d" exitCode=60 Oct 30 15:30:03.408650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.408628 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerDied","Data":"f68716082ed48c90fb1a4c6637855f36daada28202decbe03962ea1248f3238d"} Oct 30 15:30:03.409045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.408707 6554 scope.go:122] "RemoveContainer" containerID="f68716082ed48c90fb1a4c6637855f36daada28202decbe03962ea1248f3238d" Oct 30 15:30:03.410681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.410563 6554 generic.go:356] "Generic (PLEG): container finished" podID="5603e708-d420-4143-9cfa-68243d44c66c" containerID="1870a5b430c257a5f6ed1b0883dd77a10d93b6781c4c6599d7c33ab54405260d" exitCode=60 Oct 30 15:30:03.410681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.410621 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerDied","Data":"1870a5b430c257a5f6ed1b0883dd77a10d93b6781c4c6599d7c33ab54405260d"} Oct 30 15:30:03.410856 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:03.410840 6554 scope.go:122] "RemoveContainer" containerID="1870a5b430c257a5f6ed1b0883dd77a10d93b6781c4c6599d7c33ab54405260d" Oct 30 15:30:04.420405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.420368 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerStarted","Data":"69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d"} Oct 30 15:30:04.422903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.422873 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerStarted","Data":"703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1"} Oct 30 15:30:04.424830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.424802 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerStarted","Data":"4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e"} Oct 30 15:30:04.426854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.426829 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerStarted","Data":"a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10"} Oct 30 15:30:04.428519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.428490 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerStarted","Data":"d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163"} Oct 30 15:30:04.435647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.435616 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerStarted","Data":"b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9"} Oct 30 15:30:04.437778 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.437754 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerStarted","Data":"afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a"} Oct 30 15:30:04.440732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.440708 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerStarted","Data":"310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29"} Oct 30 15:30:04.442534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:04.442513 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerStarted","Data":"f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80"} Oct 30 15:30:05.484833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.483523 6554 generic.go:356] "Generic (PLEG): container finished" podID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerID="4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e" exitCode=60 Oct 30 15:30:05.484833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.483602 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerDied","Data":"4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e"} Oct 30 15:30:05.484833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.483666 6554 scope.go:122] "RemoveContainer" containerID="0680f14a7a603e0b09abe7c975a6d7f8cf5dfef8d7785ecbab8faf155626158f" Oct 30 15:30:05.484833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.483958 6554 scope.go:122] "RemoveContainer" containerID="4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e" Oct 30 15:30:05.484833 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.484288 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt_odh-ai-gateway-infra(533a44a7-a4f0-4561-bf06-7d827d914f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" Oct 30 15:30:05.487790 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.487759 6554 generic.go:356] "Generic (PLEG): container finished" podID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerID="a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10" exitCode=60 Oct 30 15:30:05.487919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.487841 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerDied","Data":"a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10"} Oct 30 15:30:05.488395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.488086 6554 scope.go:122] "RemoveContainer" containerID="a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10" Oct 30 15:30:05.488779 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.488755 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns_odh-ai-gateway-infra(b46e1e22-4cdb-424f-82ec-2e12568a543f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" Oct 30 15:30:05.490526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.490502 6554 generic.go:356] "Generic (PLEG): container finished" podID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerID="d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163" exitCode=60 Oct 30 15:30:05.490610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.490575 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerDied","Data":"d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163"} Oct 30 15:30:05.490833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.490818 6554 scope.go:122] "RemoveContainer" containerID="d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163" Oct 30 15:30:05.491440 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.491394 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt_odh-ai-gateway-infra(64e89686-0c6d-4fd5-9f6d-56a4a4d01c36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" Oct 30 15:30:05.493036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.493012 6554 generic.go:356] "Generic (PLEG): container finished" podID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerID="b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9" exitCode=60 Oct 30 15:30:05.493114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.493098 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerDied","Data":"b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9"} Oct 30 15:30:05.493450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.493433 6554 scope.go:122] "RemoveContainer" containerID="b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9" Oct 30 15:30:05.493699 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.493680 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c_odh-ai-gateway-infra(def945da-3b2c-495f-b07f-7ec7cd6b8d39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" Oct 30 15:30:05.495635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.495615 6554 generic.go:356] "Generic (PLEG): container finished" podID="261ecca1-0870-48a2-b233-5364600435b4" containerID="afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a" exitCode=60 Oct 30 15:30:05.495728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.495680 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerDied","Data":"afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a"} Oct 30 15:30:05.495889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.495876 6554 scope.go:122] "RemoveContainer" containerID="afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a" Oct 30 15:30:05.496183 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.496160 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft_odh-ai-gateway-infra(261ecca1-0870-48a2-b233-5364600435b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" podUID="261ecca1-0870-48a2-b233-5364600435b4" Oct 30 15:30:05.497808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.497787 6554 generic.go:356] "Generic (PLEG): container finished" podID="66217a85-3544-4350-8c0b-a4873f0b2962" containerID="310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29" exitCode=60 Oct 30 15:30:05.497900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.497817 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerDied","Data":"310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29"} Oct 30 15:30:05.498102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.498084 6554 scope.go:122] "RemoveContainer" containerID="310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29" Oct 30 15:30:05.498418 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.498338 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf_odh-ai-gateway-infra(66217a85-3544-4350-8c0b-a4873f0b2962)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" Oct 30 15:30:05.499782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.499764 6554 generic.go:356] "Generic (PLEG): container finished" podID="5603e708-d420-4143-9cfa-68243d44c66c" containerID="f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80" exitCode=60 Oct 30 15:30:05.499863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.499785 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerDied","Data":"f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80"} Oct 30 15:30:05.500080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.500061 6554 scope.go:122] "RemoveContainer" containerID="f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80" Oct 30 15:30:05.500362 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.500321 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl_odh-ai-gateway-infra(5603e708-d420-4143-9cfa-68243d44c66c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" podUID="5603e708-d420-4143-9cfa-68243d44c66c" Oct 30 15:30:05.501232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.501215 6554 scope.go:122] "RemoveContainer" containerID="95e8525f21d36aff7732dd45f1ad5ea5b9bd4b07081557fff310021cfdfe62a9" Oct 30 15:30:05.501748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.501727 6554 generic.go:356] "Generic (PLEG): container finished" podID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerID="69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d" exitCode=60 Oct 30 15:30:05.501836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.501796 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerDied","Data":"69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d"} Oct 30 15:30:05.502118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.502101 6554 scope.go:122] "RemoveContainer" containerID="69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d" Oct 30 15:30:05.502388 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.502364 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4_odh-ai-gateway-infra(2f1cd589-4fe8-4a98-a633-0129d9d4bbb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" Oct 30 15:30:05.503792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.503772 6554 generic.go:356] "Generic (PLEG): container finished" podID="8a856bbd-fcbd-4205-be12-58918509137d" containerID="703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1" exitCode=60 Oct 30 15:30:05.503866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.503847 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerDied","Data":"703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1"} Oct 30 15:30:05.504126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.504112 6554 scope.go:122] "RemoveContainer" containerID="703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1" Oct 30 15:30:05.504393 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:05.504375 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889570-rxvnj_odh-ai-gateway-infra(8a856bbd-fcbd-4205-be12-58918509137d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" podUID="8a856bbd-fcbd-4205-be12-58918509137d" Oct 30 15:30:05.514815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.514795 6554 scope.go:122] "RemoveContainer" containerID="772763d83934f48c41bea615deaf8b4aba0f9e0468288650363402e0d6527a6a" Oct 30 15:30:05.528320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.528298 6554 scope.go:122] "RemoveContainer" containerID="6b4547a4cf375a1322ea268d6136978707517d67c274ab1f954a49d1cb24fcf9" Oct 30 15:30:05.548341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.548319 6554 scope.go:122] "RemoveContainer" containerID="dbff1c72f2a9b8a3e1d81dd6edd8741eb701fbc4584ce01298a8a1a4dcad78d3" Oct 30 15:30:05.589092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.589065 6554 scope.go:122] "RemoveContainer" containerID="f68716082ed48c90fb1a4c6637855f36daada28202decbe03962ea1248f3238d" Oct 30 15:30:05.604060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.604039 6554 scope.go:122] "RemoveContainer" containerID="1870a5b430c257a5f6ed1b0883dd77a10d93b6781c4c6599d7c33ab54405260d" Oct 30 15:30:05.618892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.618864 6554 scope.go:122] "RemoveContainer" containerID="553f7ffa4df05c8f38688dfed4624a881a3d08fe82038c8469f1d0c078461d15" Oct 30 15:30:05.632310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:05.632290 6554 scope.go:122] "RemoveContainer" containerID="1f2e834271aa8fbe2097789dd20078680d31469655fe103582756ae5eacba2bd" Oct 30 15:30:06.514186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.514157 6554 scope.go:122] "RemoveContainer" containerID="f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80" Oct 30 15:30:06.514638 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.514481 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl_odh-ai-gateway-infra(5603e708-d420-4143-9cfa-68243d44c66c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" podUID="5603e708-d420-4143-9cfa-68243d44c66c" Oct 30 15:30:06.516244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.516221 6554 scope.go:122] "RemoveContainer" containerID="69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d" Oct 30 15:30:06.516529 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.516505 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4_odh-ai-gateway-infra(2f1cd589-4fe8-4a98-a633-0129d9d4bbb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" Oct 30 15:30:06.518159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.518130 6554 scope.go:122] "RemoveContainer" containerID="703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1" Oct 30 15:30:06.518421 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.518397 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889570-rxvnj_odh-ai-gateway-infra(8a856bbd-fcbd-4205-be12-58918509137d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" podUID="8a856bbd-fcbd-4205-be12-58918509137d" Oct 30 15:30:06.519829 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.519801 6554 scope.go:122] "RemoveContainer" containerID="4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e" Oct 30 15:30:06.520060 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.520038 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt_odh-ai-gateway-infra(533a44a7-a4f0-4561-bf06-7d827d914f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" Oct 30 15:30:06.521390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.521368 6554 scope.go:122] "RemoveContainer" containerID="a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10" Oct 30 15:30:06.521614 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.521593 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns_odh-ai-gateway-infra(b46e1e22-4cdb-424f-82ec-2e12568a543f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" Oct 30 15:30:06.523023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.523003 6554 scope.go:122] "RemoveContainer" containerID="d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163" Oct 30 15:30:06.523256 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.523224 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt_odh-ai-gateway-infra(64e89686-0c6d-4fd5-9f6d-56a4a4d01c36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" Oct 30 15:30:06.524669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.524652 6554 scope.go:122] "RemoveContainer" containerID="b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9" Oct 30 15:30:06.524893 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.524874 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c_odh-ai-gateway-infra(def945da-3b2c-495f-b07f-7ec7cd6b8d39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" Oct 30 15:30:06.526274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.526254 6554 scope.go:122] "RemoveContainer" containerID="afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a" Oct 30 15:30:06.526510 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.526487 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft_odh-ai-gateway-infra(261ecca1-0870-48a2-b233-5364600435b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" podUID="261ecca1-0870-48a2-b233-5364600435b4" Oct 30 15:30:06.527824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:06.527809 6554 scope.go:122] "RemoveContainer" containerID="310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29" Oct 30 15:30:06.528000 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:06.527983 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf_odh-ai-gateway-infra(66217a85-3544-4350-8c0b-a4873f0b2962)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" Oct 30 15:30:07.432554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:07.432511 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:30:07.432742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:07.432581 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:30:09.202898 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:09.202859 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:30:10.567845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:10.567770 6554 generic.go:356] "Generic (PLEG): container finished" podID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerID="646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2" exitCode=0 Oct 30 15:30:10.567845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:10.567814 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" event={"ID":"53a97a21-862a-4156-9ffb-8dfadc5b306c","Type":"ContainerDied","Data":"646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2"} Oct 30 15:30:10.568300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:10.568128 6554 scope.go:122] "RemoveContainer" containerID="646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2" Oct 30 15:30:10.569707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:10.569677 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:30:10.569779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:10.569749 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:30:10.569821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:10.569781 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:30:11.591918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:11.591883 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" event={"ID":"53a97a21-862a-4156-9ffb-8dfadc5b306c","Type":"ContainerStarted","Data":"df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975"} Oct 30 15:30:12.149774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:12.149735 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 15:30:12.149985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:12.149785 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 15:30:12.180511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:12.180480 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:30:13.150130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:13.150093 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 15:30:13.150596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:13.150160 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 15:30:14.149205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:14.149168 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 15:30:14.149393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:14.149229 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 15:30:15.149688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:15.149655 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 15:30:15.150086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:15.149705 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 15:30:16.152314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:16.152282 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body= Oct 30 15:30:16.152750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:16.152336 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" Oct 30 15:30:16.152750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:16.152543 6554 scope.go:122] "RemoveContainer" containerID="69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d" Oct 30 15:30:16.811807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:16.811775 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerStarted","Data":"cfea536dfec0bc3b39369f8ea1d2529302b50303c24f3f6dd85eeefa672d08f6"} Oct 30 15:30:17.147079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.147043 6554 scope.go:122] "RemoveContainer" containerID="4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e" Oct 30 15:30:17.147296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.147253 6554 scope.go:122] "RemoveContainer" containerID="b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9" Oct 30 15:30:17.147656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.147633 6554 scope.go:122] "RemoveContainer" containerID="f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80" Oct 30 15:30:17.148100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.148081 6554 scope.go:122] "RemoveContainer" containerID="afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a" Oct 30 15:30:17.189909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.189270 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9"] Oct 30 15:30:17.199032 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.199008 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.213657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.213624 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9"] Oct 30 15:30:17.225868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.218444 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-a1654b\"" Oct 30 15:30:17.225868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.220482 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-a1654b-dockercfg-w7q98\"" Oct 30 15:30:17.316905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.313460 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:30:17.539591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.539539 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc"] Oct 30 15:30:17.560724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.560687 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc"] Oct 30 15:30:17.560908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.560752 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:30:17.560908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.560820 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:30:17.561053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.560911 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.598699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.598670 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46tr\" (UniqueName: \"kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr\") pod \"payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.598810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.598723 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.700442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.700404 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ss27\" (UniqueName: \"kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27\") pod \"payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.700658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.700629 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.700832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.700800 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46tr\" (UniqueName: \"kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr\") pod \"payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.700915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.700900 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.701968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.701941 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.727020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.726950 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46tr\" (UniqueName: \"kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr\") pod \"payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.803470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.802127 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ss27\" (UniqueName: \"kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27\") pod \"payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.803470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.802307 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.803470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.803073 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.820911 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.820881 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerStarted","Data":"b6adadfbda2bce3cc528c129656e78275912f1673add153afcf33bfff09a1bdf"} Oct 30 15:30:17.822847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.822826 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerStarted","Data":"3aa5e2d86e6454b6c504372054457522e74aff3579645b22ca8fb850176a7245"} Oct 30 15:30:17.824723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.824703 6554 generic.go:356] "Generic (PLEG): container finished" podID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerID="cfea536dfec0bc3b39369f8ea1d2529302b50303c24f3f6dd85eeefa672d08f6" exitCode=60 Oct 30 15:30:17.824835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.824773 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerDied","Data":"cfea536dfec0bc3b39369f8ea1d2529302b50303c24f3f6dd85eeefa672d08f6"} Oct 30 15:30:17.824907 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.824849 6554 scope.go:122] "RemoveContainer" containerID="69e49226a805627bc33ad06acb3315029eaa72a00819472832171c0597d4344d" Oct 30 15:30:17.825115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.825097 6554 scope.go:122] "RemoveContainer" containerID="cfea536dfec0bc3b39369f8ea1d2529302b50303c24f3f6dd85eeefa672d08f6" Oct 30 15:30:17.825396 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:17.825341 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4_odh-ai-gateway-infra(2f1cd589-4fe8-4a98-a633-0129d9d4bbb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" Oct 30 15:30:17.826971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.826952 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerStarted","Data":"50f1a97c74566bdf5204b240e634f0c4b32d526064547a958080b9e38143ae9c"} Oct 30 15:30:17.827731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.827710 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ss27\" (UniqueName: \"kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27\") pod \"payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:17.866074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.866051 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:17.867462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.867443 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4"] Oct 30 15:30:17.897877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:17.897852 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:18.148371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.147439 6554 scope.go:122] "RemoveContainer" containerID="a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10" Oct 30 15:30:18.148371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.147751 6554 scope.go:122] "RemoveContainer" containerID="703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1" Oct 30 15:30:18.148371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.148211 6554 scope.go:122] "RemoveContainer" containerID="310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29" Oct 30 15:30:18.150399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.148797 6554 scope.go:122] "RemoveContainer" containerID="d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163" Oct 30 15:30:18.242439 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:30:18.242414 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbc1fbb_13b4_4151_b7e0_23cf57b0e940.slice/crio-29a0d67b99624a15c11e80a0b29fa0ec1645001e63758d6bdf3785b702458ab3 WatchSource:0}: Error finding container 29a0d67b99624a15c11e80a0b29fa0ec1645001e63758d6bdf3785b702458ab3: Status 404 returned error can't find the container with id 29a0d67b99624a15c11e80a0b29fa0ec1645001e63758d6bdf3785b702458ab3 Oct 30 15:30:18.243166 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.242992 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9"] Oct 30 15:30:18.285793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.285769 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc"] Oct 30 15:30:18.362425 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:30:18.361792 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eda1b3c_8310_46b5_a4dd_e89bcc886869.slice/crio-4883fd352743078498ef96234eaf7228e5168409b8a23e16d615ced15107e669 WatchSource:0}: Error finding container 4883fd352743078498ef96234eaf7228e5168409b8a23e16d615ced15107e669: Status 404 returned error can't find the container with id 4883fd352743078498ef96234eaf7228e5168409b8a23e16d615ced15107e669 Oct 30 15:30:18.839706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.839610 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" event={"ID":"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940","Type":"ContainerStarted","Data":"20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557"} Oct 30 15:30:18.839706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.839680 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" event={"ID":"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940","Type":"ContainerStarted","Data":"29a0d67b99624a15c11e80a0b29fa0ec1645001e63758d6bdf3785b702458ab3"} Oct 30 15:30:18.840048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.839830 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:18.843860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.843817 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerStarted","Data":"ae49fbfeb62ef40128479b4b65abdfbede961327f4d212cc37330633bbd4983a"} Oct 30 15:30:18.846222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.846195 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" event={"ID":"7eda1b3c-8310-46b5-a4dd-e89bcc886869","Type":"ContainerStarted","Data":"379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0"} Oct 30 15:30:18.846444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.846255 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" event={"ID":"7eda1b3c-8310-46b5-a4dd-e89bcc886869","Type":"ContainerStarted","Data":"4883fd352743078498ef96234eaf7228e5168409b8a23e16d615ced15107e669"} Oct 30 15:30:18.846444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.846289 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:18.848502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.848480 6554 generic.go:356] "Generic (PLEG): container finished" podID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerID="50f1a97c74566bdf5204b240e634f0c4b32d526064547a958080b9e38143ae9c" exitCode=60 Oct 30 15:30:18.848619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.848553 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerDied","Data":"50f1a97c74566bdf5204b240e634f0c4b32d526064547a958080b9e38143ae9c"} Oct 30 15:30:18.848619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.848611 6554 scope.go:122] "RemoveContainer" containerID="4d979ee8891cfed95aeeb6e9c43930bba3ab57d263cd7f80bdee87384adf6f3e" Oct 30 15:30:18.848887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.848870 6554 scope.go:122] "RemoveContainer" containerID="50f1a97c74566bdf5204b240e634f0c4b32d526064547a958080b9e38143ae9c" Oct 30 15:30:18.849187 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:18.849162 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt_odh-ai-gateway-infra(533a44a7-a4f0-4561-bf06-7d827d914f13)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" Oct 30 15:30:18.850887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.850734 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerStarted","Data":"4b628065aa6f5306bfd4ccc3238d07477709cc20fa4f05819591cf9eac8be2ad"} Oct 30 15:30:18.852988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.852967 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerStarted","Data":"c4d50bc8a514c9ffda7d9bf70b689d5e882f8ea7de994edcc81e16952674f3b3"} Oct 30 15:30:18.860508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.859814 6554 generic.go:356] "Generic (PLEG): container finished" podID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerID="b6adadfbda2bce3cc528c129656e78275912f1673add153afcf33bfff09a1bdf" exitCode=60 Oct 30 15:30:18.860508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.859888 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerDied","Data":"b6adadfbda2bce3cc528c129656e78275912f1673add153afcf33bfff09a1bdf"} Oct 30 15:30:18.860508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.860204 6554 scope.go:122] "RemoveContainer" containerID="b6adadfbda2bce3cc528c129656e78275912f1673add153afcf33bfff09a1bdf" Oct 30 15:30:18.860508 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:18.860463 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c_odh-ai-gateway-infra(def945da-3b2c-495f-b07f-7ec7cd6b8d39)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" Oct 30 15:30:18.861985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.861966 6554 generic.go:356] "Generic (PLEG): container finished" podID="261ecca1-0870-48a2-b233-5364600435b4" containerID="3aa5e2d86e6454b6c504372054457522e74aff3579645b22ca8fb850176a7245" exitCode=60 Oct 30 15:30:18.862096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.862032 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerDied","Data":"3aa5e2d86e6454b6c504372054457522e74aff3579645b22ca8fb850176a7245"} Oct 30 15:30:18.862312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.862297 6554 scope.go:122] "RemoveContainer" containerID="3aa5e2d86e6454b6c504372054457522e74aff3579645b22ca8fb850176a7245" Oct 30 15:30:18.862544 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:18.862525 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft_odh-ai-gateway-infra(261ecca1-0870-48a2-b233-5364600435b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" podUID="261ecca1-0870-48a2-b233-5364600435b4" Oct 30 15:30:18.864053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.864029 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerStarted","Data":"3994849195dd90773e02c9f198dd0a846e02bccdb3f04063af77764604761c99"} Oct 30 15:30:18.865924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.865890 6554 scope.go:122] "RemoveContainer" containerID="b556e29626c0d1ca2376c68030a8ecff5006af8e1a4295800bb1d52749e888d9" Oct 30 15:30:18.866026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.866006 6554 generic.go:356] "Generic (PLEG): container finished" podID="5603e708-d420-4143-9cfa-68243d44c66c" containerID="fbefd15261c72b4ba93c839cabe7f2461efc28ba14347dff45645a089f595d06" exitCode=60 Oct 30 15:30:18.866133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.866034 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerDied","Data":"fbefd15261c72b4ba93c839cabe7f2461efc28ba14347dff45645a089f595d06"} Oct 30 15:30:18.866850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.866829 6554 scope.go:122] "RemoveContainer" containerID="fbefd15261c72b4ba93c839cabe7f2461efc28ba14347dff45645a089f595d06" Oct 30 15:30:18.867058 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:18.867041 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl_odh-ai-gateway-infra(5603e708-d420-4143-9cfa-68243d44c66c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" podUID="5603e708-d420-4143-9cfa-68243d44c66c" Oct 30 15:30:18.884651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.884044 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c"] Oct 30 15:30:18.885822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.885794 6554 scope.go:122] "RemoveContainer" containerID="afe492a4f043d7ce8dec6481f711a04a807853d649c42e71f4ad7be1f221145a" Oct 30 15:30:18.887777 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.886926 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" podStartSLOduration=1.88690858 podStartE2EDuration="1.88690858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:18.884397138 +0000 UTC m=+2522.843438544" watchObservedRunningTime="2026-10-30 15:30:18.88690858 +0000 UTC m=+2522.845949987" Oct 30 15:30:18.906912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.906890 6554 scope.go:122] "RemoveContainer" containerID="f0806dd613d84b26f071950a93fb7af1fa9e65a2b21ca771ecc2bf7d029bcd80" Oct 30 15:30:18.930977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:18.930946 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt"] Oct 30 15:30:19.009925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.009900 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:19.102524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.102440 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft"] Oct 30 15:30:19.126688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.126658 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd\" (UniqueName: \"kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd\") pod \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " Oct 30 15:30:19.126868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.126826 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca\") pod \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\" (UID: \"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1\") " Oct 30 15:30:19.127242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.127216 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca" pod "2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" (UID: "2f1cd589-4fe8-4a98-a633-0129d9d4bbb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:19.127643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.127619 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:19.129095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.129065 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd" pod "2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" (UID: "2f1cd589-4fe8-4a98-a633-0129d9d4bbb1"). InnerVolumeSpecName "kube-api-access-jdhkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:19.180854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.180802 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" podStartSLOduration=2.180786378 podStartE2EDuration="2.180786378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:19.179050925 +0000 UTC m=+2523.138092341" watchObservedRunningTime="2026-10-30 15:30:19.180786378 +0000 UTC m=+2523.139827784" Oct 30 15:30:19.228649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.228618 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdhkd\" (UniqueName: \"kubernetes.io/projected/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1-kube-api-access-jdhkd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:19.876088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.876056 6554 generic.go:356] "Generic (PLEG): container finished" podID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerID="4b628065aa6f5306bfd4ccc3238d07477709cc20fa4f05819591cf9eac8be2ad" exitCode=60 Oct 30 15:30:19.876541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.876128 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerDied","Data":"4b628065aa6f5306bfd4ccc3238d07477709cc20fa4f05819591cf9eac8be2ad"} Oct 30 15:30:19.876541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.876183 6554 scope.go:122] "RemoveContainer" containerID="a8d3df7540645edcdf6eceedbe7c67ee529482e4acd3f12faf00bd7ec0c30b10" Oct 30 15:30:19.876676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.876565 6554 scope.go:122] "RemoveContainer" containerID="4b628065aa6f5306bfd4ccc3238d07477709cc20fa4f05819591cf9eac8be2ad" Oct 30 15:30:19.876904 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:19.876877 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns_odh-ai-gateway-infra(b46e1e22-4cdb-424f-82ec-2e12568a543f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" Oct 30 15:30:19.878316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.878296 6554 generic.go:356] "Generic (PLEG): container finished" podID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerID="c4d50bc8a514c9ffda7d9bf70b689d5e882f8ea7de994edcc81e16952674f3b3" exitCode=60 Oct 30 15:30:19.878420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.878378 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerDied","Data":"c4d50bc8a514c9ffda7d9bf70b689d5e882f8ea7de994edcc81e16952674f3b3"} Oct 30 15:30:19.878762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.878740 6554 scope.go:122] "RemoveContainer" containerID="c4d50bc8a514c9ffda7d9bf70b689d5e882f8ea7de994edcc81e16952674f3b3" Oct 30 15:30:19.879053 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:19.879034 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt_odh-ai-gateway-infra(64e89686-0c6d-4fd5-9f6d-56a4a4d01c36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" Oct 30 15:30:19.884281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.884262 6554 generic.go:356] "Generic (PLEG): container finished" podID="66217a85-3544-4350-8c0b-a4873f0b2962" containerID="3994849195dd90773e02c9f198dd0a846e02bccdb3f04063af77764604761c99" exitCode=60 Oct 30 15:30:19.884410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.884333 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerDied","Data":"3994849195dd90773e02c9f198dd0a846e02bccdb3f04063af77764604761c99"} Oct 30 15:30:19.884805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.884716 6554 scope.go:122] "RemoveContainer" containerID="3994849195dd90773e02c9f198dd0a846e02bccdb3f04063af77764604761c99" Oct 30 15:30:19.885064 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:19.885021 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf_odh-ai-gateway-infra(66217a85-3544-4350-8c0b-a4873f0b2962)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" Oct 30 15:30:19.887699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.887678 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" Oct 30 15:30:19.887961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.887938 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4" event={"ID":"2f1cd589-4fe8-4a98-a633-0129d9d4bbb1","Type":"ContainerDied","Data":"8ef1a08ea520769ad78b61fc2af4a1717b4f58599d80de1854fe8b2470782c1b"} Oct 30 15:30:19.889801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.889779 6554 generic.go:356] "Generic (PLEG): container finished" podID="8a856bbd-fcbd-4205-be12-58918509137d" containerID="ae49fbfeb62ef40128479b4b65abdfbede961327f4d212cc37330633bbd4983a" exitCode=60 Oct 30 15:30:19.889897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.889865 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerDied","Data":"ae49fbfeb62ef40128479b4b65abdfbede961327f4d212cc37330633bbd4983a"} Oct 30 15:30:19.890257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.890234 6554 scope.go:122] "RemoveContainer" containerID="ae49fbfeb62ef40128479b4b65abdfbede961327f4d212cc37330633bbd4983a" Oct 30 15:30:19.890610 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:19.890581 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889570-rxvnj_odh-ai-gateway-infra(8a856bbd-fcbd-4205-be12-58918509137d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" podUID="8a856bbd-fcbd-4205-be12-58918509137d" Oct 30 15:30:19.895167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.895141 6554 scope.go:122] "RemoveContainer" containerID="d3d53affb0ec4b507c5dff029bf994adb2157022bb8e7608d3aad96a02f55163" Oct 30 15:30:19.911542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.911520 6554 scope.go:122] "RemoveContainer" containerID="310a7a53ee30baf3a651595adf16f04dfed8195779bc995b08b981361791fc29" Oct 30 15:30:19.930331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.930309 6554 scope.go:122] "RemoveContainer" containerID="cfea536dfec0bc3b39369f8ea1d2529302b50303c24f3f6dd85eeefa672d08f6" Oct 30 15:30:19.945259 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.945237 6554 scope.go:122] "RemoveContainer" containerID="703b39285dbb370e75d433f5748c6b5d3d9791ccaeb0686a61d5bbb1efcdfab1" Oct 30 15:30:19.945545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:19.945525 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf"] Oct 30 15:30:20.095095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.095059 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt"] Oct 30 15:30:20.095391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.095338 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:20.106662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.106633 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns"] Oct 30 15:30:20.152482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.152451 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj"] Oct 30 15:30:20.174687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.174665 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4"] Oct 30 15:30:20.174822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.174697 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7aede6-29889570-bfkb4"] Oct 30 15:30:20.178600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.178455 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k\" (UniqueName: \"kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k\") pod \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " Oct 30 15:30:20.178600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.178515 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca\") pod \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\" (UID: \"def945da-3b2c-495f-b07f-7ec7cd6b8d39\") " Oct 30 15:30:20.179163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.179132 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca" pod "def945da-3b2c-495f-b07f-7ec7cd6b8d39" (UID: "def945da-3b2c-495f-b07f-7ec7cd6b8d39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:20.182012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.181969 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k" pod "def945da-3b2c-495f-b07f-7ec7cd6b8d39" (UID: "def945da-3b2c-495f-b07f-7ec7cd6b8d39"). InnerVolumeSpecName "kube-api-access-bcv8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:20.184435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.184418 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:20.251549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.251522 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:20.279393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.279342 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f\" (UniqueName: \"kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f\") pod \"261ecca1-0870-48a2-b233-5364600435b4\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " Oct 30 15:30:20.279598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.279579 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca\") pod \"261ecca1-0870-48a2-b233-5364600435b4\" (UID: \"261ecca1-0870-48a2-b233-5364600435b4\") " Oct 30 15:30:20.279982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.279948 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca" pod "261ecca1-0870-48a2-b233-5364600435b4" (UID: "261ecca1-0870-48a2-b233-5364600435b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:20.280858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.280837 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcv8k\" (UniqueName: \"kubernetes.io/projected/def945da-3b2c-495f-b07f-7ec7cd6b8d39-kube-api-access-bcv8k\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.280858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.280857 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def945da-3b2c-495f-b07f-7ec7cd6b8d39-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.280986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.280866 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261ecca1-0870-48a2-b233-5364600435b4-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.281672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.281652 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f" pod "261ecca1-0870-48a2-b233-5364600435b4" (UID: "261ecca1-0870-48a2-b233-5364600435b4"). InnerVolumeSpecName "kube-api-access-5ch7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:20.320257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.320231 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl"] Oct 30 15:30:20.382489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.382331 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca\") pod \"533a44a7-a4f0-4561-bf06-7d827d914f13\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " Oct 30 15:30:20.382489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.382401 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8\" (UniqueName: \"kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8\") pod \"533a44a7-a4f0-4561-bf06-7d827d914f13\" (UID: \"533a44a7-a4f0-4561-bf06-7d827d914f13\") " Oct 30 15:30:20.382808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.382778 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca" pod "533a44a7-a4f0-4561-bf06-7d827d914f13" (UID: "533a44a7-a4f0-4561-bf06-7d827d914f13"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:20.383271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.383242 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533a44a7-a4f0-4561-bf06-7d827d914f13-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.383401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.383275 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ch7f\" (UniqueName: \"kubernetes.io/projected/261ecca1-0870-48a2-b233-5364600435b4-kube-api-access-5ch7f\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.385724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.385700 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8" pod "533a44a7-a4f0-4561-bf06-7d827d914f13" (UID: "533a44a7-a4f0-4561-bf06-7d827d914f13"). InnerVolumeSpecName "kube-api-access-xlrs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:20.484744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.484708 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrs8\" (UniqueName: \"kubernetes.io/projected/533a44a7-a4f0-4561-bf06-7d827d914f13-kube-api-access-xlrs8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.505289 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.505265 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:20.585256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.585231 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9\" (UniqueName: \"kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9\") pod \"5603e708-d420-4143-9cfa-68243d44c66c\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " Oct 30 15:30:20.585450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.585430 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca\") pod \"5603e708-d420-4143-9cfa-68243d44c66c\" (UID: \"5603e708-d420-4143-9cfa-68243d44c66c\") " Oct 30 15:30:20.585804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.585776 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca" pod "5603e708-d420-4143-9cfa-68243d44c66c" (UID: "5603e708-d420-4143-9cfa-68243d44c66c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:20.586171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.586152 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5603e708-d420-4143-9cfa-68243d44c66c-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.587509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.587482 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9" pod "5603e708-d420-4143-9cfa-68243d44c66c" (UID: "5603e708-d420-4143-9cfa-68243d44c66c"). InnerVolumeSpecName "kube-api-access-qrkx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:20.687945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.687918 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrkx9\" (UniqueName: \"kubernetes.io/projected/5603e708-d420-4143-9cfa-68243d44c66c-kube-api-access-qrkx9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:20.904702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.904671 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" event={"ID":"5603e708-d420-4143-9cfa-68243d44c66c","Type":"ContainerDied","Data":"e59e4b80ea641a7bdf742264215a0914fe626a0655209a3d4b6d2b9b50d309d8"} Oct 30 15:30:20.905115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.904681 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl" Oct 30 15:30:20.905115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.904764 6554 scope.go:122] "RemoveContainer" containerID="fbefd15261c72b4ba93c839cabe7f2461efc28ba14347dff45645a089f595d06" Oct 30 15:30:20.909719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.909699 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" Oct 30 15:30:20.909834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.909738 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt" event={"ID":"533a44a7-a4f0-4561-bf06-7d827d914f13","Type":"ContainerDied","Data":"387ef1899f5c23a9bf9aec7a921c694274cdc954ca51db02666253e94faed941"} Oct 30 15:30:20.919099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.919076 6554 scope.go:122] "RemoveContainer" containerID="50f1a97c74566bdf5204b240e634f0c4b32d526064547a958080b9e38143ae9c" Oct 30 15:30:20.923220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.923137 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" event={"ID":"def945da-3b2c-495f-b07f-7ec7cd6b8d39","Type":"ContainerDied","Data":"f5665c0cb2dee4db59bcd0a64dce31c35e4ce351392313290ecd36bd1b82a1d1"} Oct 30 15:30:20.923220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.923151 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c" Oct 30 15:30:20.925138 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.925048 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" event={"ID":"261ecca1-0870-48a2-b233-5364600435b4","Type":"ContainerDied","Data":"0e08f89d40bdf0ba5677a244f22845633d164c7b5db1c0810e6598fee785bf43"} Oct 30 15:30:20.925249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.925165 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft" Oct 30 15:30:20.936706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.936676 6554 scope.go:122] "RemoveContainer" containerID="b6adadfbda2bce3cc528c129656e78275912f1673add153afcf33bfff09a1bdf" Oct 30 15:30:20.963453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.963426 6554 scope.go:122] "RemoveContainer" containerID="3aa5e2d86e6454b6c504372054457522e74aff3579645b22ca8fb850176a7245" Oct 30 15:30:20.990030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.989396 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl"] Oct 30 15:30:20.998420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:20.998394 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f3a578-29889570-sklxl"] Oct 30 15:30:21.106996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.106970 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:21.155387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.155321 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:21.168817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.168790 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:21.181943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.181912 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt"] Oct 30 15:30:21.187576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.187551 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7024f9-29889570-p8jlt"] Oct 30 15:30:21.202065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.201964 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2\" (UniqueName: \"kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2\") pod \"8a856bbd-fcbd-4205-be12-58918509137d\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " Oct 30 15:30:21.202065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.202040 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca\") pod \"8a856bbd-fcbd-4205-be12-58918509137d\" (UID: \"8a856bbd-fcbd-4205-be12-58918509137d\") " Oct 30 15:30:21.202936 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.202907 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca" pod "8a856bbd-fcbd-4205-be12-58918509137d" (UID: "8a856bbd-fcbd-4205-be12-58918509137d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.204985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.204964 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2" pod "8a856bbd-fcbd-4205-be12-58918509137d" (UID: "8a856bbd-fcbd-4205-be12-58918509137d"). InnerVolumeSpecName "kube-api-access-qr6q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.225416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.225390 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c"] Oct 30 15:30:21.225536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.225439 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-83ff4e-29889570-7gm8c"] Oct 30 15:30:21.262361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.262331 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:21.303773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.303742 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58\" (UniqueName: \"kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58\") pod \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " Oct 30 15:30:21.303992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.303972 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca\") pod \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\" (UID: \"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36\") " Oct 30 15:30:21.304092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.304020 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca\") pod \"b46e1e22-4cdb-424f-82ec-2e12568a543f\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " Oct 30 15:30:21.304092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.304038 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g\" (UniqueName: \"kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g\") pod \"b46e1e22-4cdb-424f-82ec-2e12568a543f\" (UID: \"b46e1e22-4cdb-424f-82ec-2e12568a543f\") " Oct 30 15:30:21.305288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.304900 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr6q2\" (UniqueName: \"kubernetes.io/projected/8a856bbd-fcbd-4205-be12-58918509137d-kube-api-access-qr6q2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.305288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.304928 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a856bbd-fcbd-4205-be12-58918509137d-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.305288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.305201 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca" pod "64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" (UID: "64e89686-0c6d-4fd5-9f6d-56a4a4d01c36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.305288 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.305253 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca" pod "b46e1e22-4cdb-424f-82ec-2e12568a543f" (UID: "b46e1e22-4cdb-424f-82ec-2e12568a543f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.307641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.307602 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58" pod "64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" (UID: "64e89686-0c6d-4fd5-9f6d-56a4a4d01c36"). InnerVolumeSpecName "kube-api-access-qwl58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.309518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.309492 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g" pod "b46e1e22-4cdb-424f-82ec-2e12568a543f" (UID: "b46e1e22-4cdb-424f-82ec-2e12568a543f"). InnerVolumeSpecName "kube-api-access-9qg8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.315223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.315197 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft"] Oct 30 15:30:21.321023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.320999 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-05d396-29889570-gsqft"] Oct 30 15:30:21.406410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.406375 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca\") pod \"66217a85-3544-4350-8c0b-a4873f0b2962\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " Oct 30 15:30:21.406644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.406624 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc\" (UniqueName: \"kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc\") pod \"66217a85-3544-4350-8c0b-a4873f0b2962\" (UID: \"66217a85-3544-4350-8c0b-a4873f0b2962\") " Oct 30 15:30:21.407081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.406742 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca" pod "66217a85-3544-4350-8c0b-a4873f0b2962" (UID: "66217a85-3544-4350-8c0b-a4873f0b2962"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:21.407303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.407279 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.407480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.407455 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66217a85-3544-4350-8c0b-a4873f0b2962-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.407591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.407483 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46e1e22-4cdb-424f-82ec-2e12568a543f-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.407591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.407498 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qg8g\" (UniqueName: \"kubernetes.io/projected/b46e1e22-4cdb-424f-82ec-2e12568a543f-kube-api-access-9qg8g\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.407591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.407512 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwl58\" (UniqueName: \"kubernetes.io/projected/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36-kube-api-access-qwl58\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.408982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.408953 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc" pod "66217a85-3544-4350-8c0b-a4873f0b2962" (UID: "66217a85-3544-4350-8c0b-a4873f0b2962"). InnerVolumeSpecName "kube-api-access-4bzcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:21.509491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.509420 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bzcc\" (UniqueName: \"kubernetes.io/projected/66217a85-3544-4350-8c0b-a4873f0b2962-kube-api-access-4bzcc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:21.956732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.956693 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" Oct 30 15:30:21.957198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.956703 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns" event={"ID":"b46e1e22-4cdb-424f-82ec-2e12568a543f","Type":"ContainerDied","Data":"53417d6977fdf8806aedb8ce2c3aad7ca0f8a06f6916f9d12dc3ac1b23bb10cf"} Oct 30 15:30:21.957198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.956857 6554 scope.go:122] "RemoveContainer" containerID="4b628065aa6f5306bfd4ccc3238d07477709cc20fa4f05819591cf9eac8be2ad" Oct 30 15:30:21.960198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.960104 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" Oct 30 15:30:21.960198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.960145 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt" event={"ID":"64e89686-0c6d-4fd5-9f6d-56a4a4d01c36","Type":"ContainerDied","Data":"2dab19bed57903e4ab936f28230419d95565f82dde659ebe3b76d5378e2bba46"} Oct 30 15:30:21.965715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.965695 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" Oct 30 15:30:21.965838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.965732 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf" event={"ID":"66217a85-3544-4350-8c0b-a4873f0b2962","Type":"ContainerDied","Data":"72d8ae855e20be0d9662f0612bc9460648e2eed90f93ecee67a14553b574a693"} Oct 30 15:30:21.969736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.969712 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" event={"ID":"8a856bbd-fcbd-4205-be12-58918509137d","Type":"ContainerDied","Data":"47bfbcc4114fb31c22bab578f4bbd33f5b6815e1c05f12fb68fb1a82ea59f187"} Oct 30 15:30:21.970079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.969962 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj" Oct 30 15:30:21.974086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.974065 6554 scope.go:122] "RemoveContainer" containerID="c4d50bc8a514c9ffda7d9bf70b689d5e882f8ea7de994edcc81e16952674f3b3" Oct 30 15:30:21.986387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.986373 6554 scope.go:122] "RemoveContainer" containerID="3994849195dd90773e02c9f198dd0a846e02bccdb3f04063af77764604761c99" Oct 30 15:30:21.995584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:21.995565 6554 scope.go:122] "RemoveContainer" containerID="ae49fbfeb62ef40128479b4b65abdfbede961327f4d212cc37330633bbd4983a" Oct 30 15:30:22.052185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.052159 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns"] Oct 30 15:30:22.055489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.055465 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-357948-29889570-99zns"] Oct 30 15:30:22.122001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.121974 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf"] Oct 30 15:30:22.130184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.130160 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-100b2e-29889570-xr7bf"] Oct 30 15:30:22.158231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.158206 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261ecca1-0870-48a2-b233-5364600435b4" path="/var/lib/kubelet/pods/261ecca1-0870-48a2-b233-5364600435b4/volumes" Oct 30 15:30:22.158697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.158678 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" path="/var/lib/kubelet/pods/2f1cd589-4fe8-4a98-a633-0129d9d4bbb1/volumes" Oct 30 15:30:22.159130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.159114 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" path="/var/lib/kubelet/pods/533a44a7-a4f0-4561-bf06-7d827d914f13/volumes" Oct 30 15:30:22.159982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.159962 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5603e708-d420-4143-9cfa-68243d44c66c" path="/var/lib/kubelet/pods/5603e708-d420-4143-9cfa-68243d44c66c/volumes" Oct 30 15:30:22.160452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.160433 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" path="/var/lib/kubelet/pods/66217a85-3544-4350-8c0b-a4873f0b2962/volumes" Oct 30 15:30:22.160842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.160828 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" path="/var/lib/kubelet/pods/b46e1e22-4cdb-424f-82ec-2e12568a543f/volumes" Oct 30 15:30:22.161512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.161498 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" path="/var/lib/kubelet/pods/def945da-3b2c-495f-b07f-7ec7cd6b8d39/volumes" Oct 30 15:30:22.163208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.163188 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt"] Oct 30 15:30:22.169523 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.169503 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-416d92-29889570-czgzt"] Oct 30 15:30:22.208695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.208640 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj"] Oct 30 15:30:22.220109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:22.220087 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889570-rxvnj"] Oct 30 15:30:24.155984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:24.155952 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" path="/var/lib/kubelet/pods/64e89686-0c6d-4fd5-9f6d-56a4a4d01c36/volumes" Oct 30 15:30:24.156410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:24.156392 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a856bbd-fcbd-4205-be12-58918509137d" path="/var/lib/kubelet/pods/8a856bbd-fcbd-4205-be12-58918509137d/volumes" Oct 30 15:30:29.898725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:29.898645 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:30:29.898725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:29.898713 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:30:39.203858 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:39.203817 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:30:39.811448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.811416 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v"] Oct 30 15:30:39.812125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812111 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812125 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812139 6554 cpu_manager.go:414] "containerMap: removing container" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812145 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812154 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812159 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812167 6554 cpu_manager.go:414] "containerMap: removing container" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.812174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812172 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812180 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812185 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812192 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812197 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812215 6554 cpu_manager.go:414] "containerMap: removing container" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812219 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812226 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812230 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812239 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812243 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812249 6554 cpu_manager.go:414] "containerMap: removing container" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812253 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812261 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812265 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812276 6554 cpu_manager.go:414] "containerMap: removing container" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812280 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812291 6554 cpu_manager.go:414] "containerMap: removing container" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812296 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812304 6554 cpu_manager.go:414] "containerMap: removing container" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812308 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812316 6554 cpu_manager.go:414] "containerMap: removing container" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812321 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812332 6554 cpu_manager.go:414] "containerMap: removing container" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812338 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812360 6554 cpu_manager.go:414] "containerMap: removing container" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812368 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812378 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812384 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812394 6554 cpu_manager.go:414] "containerMap: removing container" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:39.812425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812399 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812505 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812512 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812520 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812525 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812532 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812539 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="533a44a7-a4f0-4561-bf06-7d827d914f13" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812545 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812551 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812556 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812562 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812568 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812573 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812580 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812585 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812590 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812595 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812601 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812608 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812614 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812619 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812626 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:39.813325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.812632 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:39.818954 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.818934 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:39.845991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.845966 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e5423a\"" Oct 30 15:30:39.851017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.850992 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v"] Oct 30 15:30:39.859743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:39.859722 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e5423a-dockercfg-qlwtf\"" Oct 30 15:30:40.014083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.014046 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbgdr\" (UniqueName: \"kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.014630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.014606 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.042374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.041952 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7"] Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043110 6554 cpu_manager.go:414] "containerMap: removing container" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043133 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a856bbd-fcbd-4205-be12-58918509137d" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043155 6554 cpu_manager.go:414] "containerMap: removing container" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043164 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043212 6554 cpu_manager.go:414] "containerMap: removing container" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043220 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043238 6554 cpu_manager.go:414] "containerMap: removing container" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043246 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043260 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043267 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1cd589-4fe8-4a98-a633-0129d9d4bbb1" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043280 6554 cpu_manager.go:414] "containerMap: removing container" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043287 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043304 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043311 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46e1e22-4cdb-424f-82ec-2e12568a543f" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043320 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043328 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043547 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="5603e708-d420-4143-9cfa-68243d44c66c" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043565 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e89686-0c6d-4fd5-9f6d-56a4a4d01c36" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043587 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="261ecca1-0870-48a2-b233-5364600435b4" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043597 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="def945da-3b2c-495f-b07f-7ec7cd6b8d39" containerName="cleanup" Oct 30 15:30:40.045360 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.043608 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66217a85-3544-4350-8c0b-a4873f0b2962" containerName="cleanup" Oct 30 15:30:40.086000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.084880 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7"] Oct 30 15:30:40.086000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.085005 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.116612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.116558 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbgdr\" (UniqueName: \"kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.116825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.116701 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.117521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.117500 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.144030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.143999 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbgdr\" (UniqueName: \"kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr\") pod \"payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.218095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.218061 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.218625 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.218266 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t47cw\" (UniqueName: \"kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw\") pod \"payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.319619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.319590 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t47cw\" (UniqueName: \"kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw\") pod \"payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.319815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.319719 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.320334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.320314 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.344930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.344874 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t47cw\" (UniqueName: \"kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw\") pod \"payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.416396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.416363 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:40.434472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.434184 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:40.819371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.817399 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7"] Oct 30 15:30:40.863903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:40.863279 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v"] Oct 30 15:30:41.234234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.234196 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr"] Oct 30 15:30:41.237913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.237877 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" event={"ID":"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1","Type":"ContainerStarted","Data":"b2244c624d08145417d2f047a924df5e8e815156c68ce287b6bbfc5f05d985aa"} Oct 30 15:30:41.238220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.237937 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" event={"ID":"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1","Type":"ContainerStarted","Data":"228046e8f2a5ecc87530981c17a6df8928917f5f652dd747b2657740e8b12897"} Oct 30 15:30:41.238220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.237992 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:41.242703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.242675 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" event={"ID":"e07f6acc-8ab7-4c60-96ce-1281f584a867","Type":"ContainerStarted","Data":"8de0af4f85ff6152240a96a9885f73e7c39a58300e2edf58256fb63278030d60"} Oct 30 15:30:41.243049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.243025 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" event={"ID":"e07f6acc-8ab7-4c60-96ce-1281f584a867","Type":"ContainerStarted","Data":"5d5d8817a9bcae22ee2f309d07307fe4985694365e11b0b3ef80e2c6a0abecbc"} Oct 30 15:30:41.243228 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.243212 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:41.339616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.339555 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" podStartSLOduration=2.339538303 podStartE2EDuration="2.339538303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:41.335659797 +0000 UTC m=+2545.294701207" watchObservedRunningTime="2026-10-30 15:30:41.339538303 +0000 UTC m=+2545.298579707" Oct 30 15:30:41.381774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:41.381718 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" podStartSLOduration=2.381702672 podStartE2EDuration="2.381702672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:30:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:30:41.379766773 +0000 UTC m=+2545.338808178" watchObservedRunningTime="2026-10-30 15:30:41.381702672 +0000 UTC m=+2545.340744076" Oct 30 15:30:46.235167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.235126 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" podUID="b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" containerName="payload-pre-processing" containerID="cri-o://2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb" gracePeriod=25 Oct 30 15:30:46.593092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.592739 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:30:46.692077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.692038 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg\" (UniqueName: \"kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg\") pod \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " Oct 30 15:30:46.692269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.692188 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume\") pod \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\" (UID: \"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224\") " Oct 30 15:30:46.692704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.692668 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume" pod "b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" (UID: "b51ab9aa-7289-40d4-9a6d-8b1d0cb39224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:30:46.693027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.693005 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:46.694219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.694188 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg" pod "b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" (UID: "b51ab9aa-7289-40d4-9a6d-8b1d0cb39224"). InnerVolumeSpecName "kube-api-access-kxmtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:30:46.793973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:46.793902 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxmtg\" (UniqueName: \"kubernetes.io/projected/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224-kube-api-access-kxmtg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:30:47.338743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.338708 6554 generic.go:356] "Generic (PLEG): container finished" podID="b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" containerID="2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb" exitCode=0 Oct 30 15:30:47.339239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.338759 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" Oct 30 15:30:47.339239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.338800 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" event={"ID":"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224","Type":"ContainerDied","Data":"2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb"} Oct 30 15:30:47.339239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.338860 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr" event={"ID":"b51ab9aa-7289-40d4-9a6d-8b1d0cb39224","Type":"ContainerDied","Data":"d94c74879cd8d361e0004475d6c1e556fec096e3a5e0ea8594ac43f711a93242"} Oct 30 15:30:47.339239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.338887 6554 scope.go:122] "RemoveContainer" containerID="2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb" Oct 30 15:30:47.349946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.349923 6554 scope.go:122] "RemoveContainer" containerID="2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb" Oct 30 15:30:47.350193 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:30:47.350172 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb\": container with ID starting with 2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb not found: ID does not exist" containerID="2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb" Oct 30 15:30:47.350246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.350202 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb"} err="failed to get container status \"2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb\": rpc error: code = NotFound desc = could not find container \"2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb\": container with ID starting with 2d736f61bedbc1b91ebb1023f4de38dbb0fa90ab70b656957e8d11ebb8b26ebb not found: ID does not exist" Oct 30 15:30:47.424599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.424560 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr"] Oct 30 15:30:47.429506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:47.429479 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf437b-5b488496f5-hvbkr"] Oct 30 15:30:48.156523 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:48.156492 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" path="/var/lib/kubelet/pods/b51ab9aa-7289-40d4-9a6d-8b1d0cb39224/volumes" Oct 30 15:30:52.253143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:52.253115 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:30:52.253704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:52.253192 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:30:53.392585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:53.392554 6554 generic.go:356] "Generic (PLEG): container finished" podID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerID="7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df" exitCode=0 Oct 30 15:30:53.393011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:53.392623 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" event={"ID":"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889","Type":"ContainerDied","Data":"7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df"} Oct 30 15:30:53.393011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:53.392963 6554 scope.go:122] "RemoveContainer" containerID="7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df" Oct 30 15:30:53.394562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:53.394532 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:30:53.394639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:53.394604 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:30:53.394690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:53.394638 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:30:54.402994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:54.402951 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" event={"ID":"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889","Type":"ContainerStarted","Data":"746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5"} Oct 30 15:30:55.296588 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:55.296557 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:30:55.296772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:55.296710 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 15:30:55.296772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:55.296760 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 15:30:56.297020 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:56.296993 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 15:30:56.297558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:56.297054 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 15:30:57.296573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:57.296542 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 15:30:57.296775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:57.296591 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 15:30:58.296133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:58.296096 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 15:30:58.296647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:58.296149 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 15:30:59.296619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:59.296582 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 15:30:59.297026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:30:59.296635 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 15:31:00.301807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:00.301776 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:31:00.302277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:00.302256 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:31:00.302855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:00.302835 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:31:02.131497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.131442 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9"] Oct 30 15:31:02.132163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.132148 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" containerName="payload-pre-processing" Oct 30 15:31:02.132215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.132164 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" containerName="payload-pre-processing" Oct 30 15:31:02.132318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.132308 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51ab9aa-7289-40d4-9a6d-8b1d0cb39224" containerName="payload-pre-processing" Oct 30 15:31:02.135864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.135845 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.160024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.159999 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9"] Oct 30 15:31:02.168596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.168571 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-a1654b-serving-cert\"" Oct 30 15:31:02.226531 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.226502 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.226731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.226705 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.227116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.227089 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.227225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.227133 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6qp\" (UniqueName: \"kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.340536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.329499 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.340536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.329557 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6qp\" (UniqueName: \"kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.340536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.329652 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.340536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.329796 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.340536 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:31:02.329999 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-serving-cert: secret "maas-api-e2e-shared-b-w1-a1654b-serving-cert" not found Oct 30 15:31:02.340536 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:31:02.330083 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls podName:827f20c2-e338-4eee-b752-a36145cee6ba nodeName:}" failed. No retries permitted until 2026-10-30 15:31:02.830060444 +0000 UTC m=+2566.789101847 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls") pod "maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" (UID: "827f20c2-e338-4eee-b752-a36145cee6ba") : secret "maas-api-e2e-shared-b-w1-a1654b-serving-cert" not found Oct 30 15:31:02.344065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.344026 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.361718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.361664 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.375667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.375640 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6qp\" (UniqueName: \"kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.836791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.836760 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:02.839295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:02.839269 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:03.050256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:03.050232 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:03.366573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:03.366478 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9"] Oct 30 15:31:03.478434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:03.478400 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" event={"ID":"827f20c2-e338-4eee-b752-a36145cee6ba","Type":"ContainerStarted","Data":"d6ee2083d4122d34c166704c151950f6e4b0a9af22eab39971432965af3b33ed"} Oct 30 15:31:04.487861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:04.487822 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" event={"ID":"827f20c2-e338-4eee-b752-a36145cee6ba","Type":"ContainerStarted","Data":"879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061"} Oct 30 15:31:04.488331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:04.487914 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:04.538474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:04.538429 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" podStartSLOduration=2.100525169 podStartE2EDuration="2.538414334s" totalImagesPullingTime="437.889165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:04.534452566 +0000 UTC m=+2568.493493975" watchObservedRunningTime="2026-10-30 15:31:04.538414334 +0000 UTC m=+2568.497455741" Oct 30 15:31:09.204947 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:31:09.204907 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:31:10.501872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:10.501842 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:31:39.203650 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:31:39.203604 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:31:39.367300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.367263 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv"] Oct 30 15:31:39.375128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.375099 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.404341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.404312 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv"] Oct 30 15:31:39.410715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.410680 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-78b513-openshift-default-dockercfg-tsbfm\"" Oct 30 15:31:39.443689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.443665 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.443847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.443722 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.443924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.443866 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.444008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.443922 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.444008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.443983 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.444203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.444182 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.444286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.444235 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.444343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.444307 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.444435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.444410 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnkzm\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.550433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.545636 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.563129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.563092 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.563405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.563381 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.563584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.563567 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnkzm\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.588374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.563827 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.588669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.588642 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.588836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.588820 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.588980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.588965 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.589129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.589115 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.596601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.565739 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.596794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.591676 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.596918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.586854 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.600481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.591893 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.600659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.592090 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.606418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.566627 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.624795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.624726 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnkzm\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.638313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.638238 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.639869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.639837 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo\") pod \"e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:39.710370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:39.704666 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:40.085995 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:31:40.085967 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f70536_e123_4dad_810c_b77ab92b8c04.slice/crio-32bba0e2d08ec56c7e921f1dbf79478b7f4fc1b5db889d38fcdf0261bd34477f WatchSource:0}: Error finding container 32bba0e2d08ec56c7e921f1dbf79478b7f4fc1b5db889d38fcdf0261bd34477f: Status 404 returned error can't find the container with id 32bba0e2d08ec56c7e921f1dbf79478b7f4fc1b5db889d38fcdf0261bd34477f Oct 30 15:31:40.086297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.086276 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv"] Oct 30 15:31:40.088895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.088859 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:31:40.089006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.088937 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:31:40.089006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.088983 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:31:40.783710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.783676 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" event={"ID":"01f70536-e123-4dad-810c-b77ab92b8c04","Type":"ContainerStarted","Data":"ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f"} Oct 30 15:31:40.784305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.783737 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" event={"ID":"01f70536-e123-4dad-810c-b77ab92b8c04","Type":"ContainerStarted","Data":"32bba0e2d08ec56c7e921f1dbf79478b7f4fc1b5db889d38fcdf0261bd34477f"} Oct 30 15:31:40.877625 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:40.877556 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podStartSLOduration=1.877540661 podStartE2EDuration="1.877540661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:40.876449051 +0000 UTC m=+2604.835490469" watchObservedRunningTime="2026-10-30 15:31:40.877540661 +0000 UTC m=+2604.836582067" Oct 30 15:31:41.707434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:41.707404 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:41.714248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:41.714220 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:41.791587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:41.791559 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:41.792503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:41.792484 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:31:42.607437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.607397 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6"] Oct 30 15:31:42.624640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.624607 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:42.633685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.633653 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6"] Oct 30 15:31:42.641831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.641803 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-631d84-dockercfg-4lnw5\"" Oct 30 15:31:42.641959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.641863 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-631d84\"" Oct 30 15:31:42.789373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.756473 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:42.789775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.789742 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs6k7\" (UniqueName: \"kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7\") pod \"payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:42.805456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.805423 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv"] Oct 30 15:31:42.885421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.882752 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:42.912897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.912841 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:42.913205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.913181 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs6k7\" (UniqueName: \"kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7\") pod \"payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:42.913544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.913521 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26njg\" (UniqueName: \"kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg\") pod \"payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:42.913765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.913745 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:42.914169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.914132 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv"] Oct 30 15:31:42.916252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.914998 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:42.948403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:42.948333 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs6k7\" (UniqueName: \"kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7\") pod \"payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:43.015108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.015076 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26njg\" (UniqueName: \"kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg\") pod \"payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:43.015272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.015145 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:43.016066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.016036 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:43.040571 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.040549 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26njg\" (UniqueName: \"kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg\") pod \"payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:43.228696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.228674 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:43.246374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.246340 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:43.544302 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:31:43.544272 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685fde02_9c86_4a8d_8f97_fa0981a18202.slice/crio-9523f35b364932b188676f009ca8c77a9930011673758b0ed4cecd60e769863e WatchSource:0}: Error finding container 9523f35b364932b188676f009ca8c77a9930011673758b0ed4cecd60e769863e: Status 404 returned error can't find the container with id 9523f35b364932b188676f009ca8c77a9930011673758b0ed4cecd60e769863e Oct 30 15:31:43.544583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.544561 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv"] Oct 30 15:31:43.580615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.580590 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6"] Oct 30 15:31:43.588616 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:31:43.588590 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab596006_3424_46d3_8e0d_a12e11e95d1d.slice/crio-af6c313f165fde5db7a9283c3c89e778989cd982fc5cddc9f61858edc8b74c99 WatchSource:0}: Error finding container af6c313f165fde5db7a9283c3c89e778989cd982fc5cddc9f61858edc8b74c99: Status 404 returned error can't find the container with id af6c313f165fde5db7a9283c3c89e778989cd982fc5cddc9f61858edc8b74c99 Oct 30 15:31:43.883033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.882952 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" event={"ID":"685fde02-9c86-4a8d-8f97-fa0981a18202","Type":"ContainerStarted","Data":"df8d415be5b5b504e38664c7ba2aaca94120eb5c85d88c573ce6e969f461bbc5"} Oct 30 15:31:43.883033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.883005 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" event={"ID":"685fde02-9c86-4a8d-8f97-fa0981a18202","Type":"ContainerStarted","Data":"9523f35b364932b188676f009ca8c77a9930011673758b0ed4cecd60e769863e"} Oct 30 15:31:43.883491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.883055 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:43.884769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.884746 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" event={"ID":"ab596006-3424-46d3-8e0d-a12e11e95d1d","Type":"ContainerStarted","Data":"f643782a9eed318bfadda67ceada0e335651744db692015d0d5cec8c304c2536"} Oct 30 15:31:43.884893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.884827 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:43.884893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.884869 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" event={"ID":"ab596006-3424-46d3-8e0d-a12e11e95d1d","Type":"ContainerStarted","Data":"af6c313f165fde5db7a9283c3c89e778989cd982fc5cddc9f61858edc8b74c99"} Oct 30 15:31:43.931405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.931345 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" podStartSLOduration=1.931331903 podStartE2EDuration="1.931331903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:43.930090993 +0000 UTC m=+2607.889132403" watchObservedRunningTime="2026-10-30 15:31:43.931331903 +0000 UTC m=+2607.890373307" Oct 30 15:31:43.984660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:43.984614 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" podStartSLOduration=1.9846027400000001 podStartE2EDuration="1.98460274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:43.982834559 +0000 UTC m=+2607.941875963" watchObservedRunningTime="2026-10-30 15:31:43.98460274 +0000 UTC m=+2607.943644144" Oct 30 15:31:45.780146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.780110 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486"] Oct 30 15:31:45.785723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.785698 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:45.815082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.815051 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78b513\"" Oct 30 15:31:45.815250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.815134 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-78b513-dockercfg-ccwhl\"" Oct 30 15:31:45.821644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.821615 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486"] Oct 30 15:31:45.870768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.870732 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hshk\" (UniqueName: \"kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk\") pod \"payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:45.870962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.870795 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:45.939375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.937562 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw"] Oct 30 15:31:45.982373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.974298 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:45.982373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.974735 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hshk\" (UniqueName: \"kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk\") pod \"payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:45.982373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.976519 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:45.982373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.977922 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:45.982373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:45.980683 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw"] Oct 30 15:31:46.068459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.066681 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hshk\" (UniqueName: \"kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk\") pod \"payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:46.109639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.108690 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:46.183173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.183093 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.183428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.183401 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22x5\" (UniqueName: \"kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5\") pod \"payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.286053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.286011 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22x5\" (UniqueName: \"kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5\") pod \"payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.286260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.286232 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.287394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.287294 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.311475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.311443 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22x5\" (UniqueName: \"kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5\") pod \"payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.322461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.322385 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.471641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.464874 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486"] Oct 30 15:31:46.474773 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:31:46.474740 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6d4d91_ee9f_4c45_8fa6_48c4d97bb00f.slice/crio-87b9cbe6c6d290f09dbe74a2f01543b4b456b72fb9dceef43a9fa11bf5df5cbb WatchSource:0}: Error finding container 87b9cbe6c6d290f09dbe74a2f01543b4b456b72fb9dceef43a9fa11bf5df5cbb: Status 404 returned error can't find the container with id 87b9cbe6c6d290f09dbe74a2f01543b4b456b72fb9dceef43a9fa11bf5df5cbb Oct 30 15:31:46.665988 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:31:46.665959 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ad4837_6451_4aab_9bef_a0eba0842edf.slice/crio-a163cf5d8017b29a74a14354aff78de079fbf080080ff1ed136b8e435671b1f9 WatchSource:0}: Error finding container a163cf5d8017b29a74a14354aff78de079fbf080080ff1ed136b8e435671b1f9: Status 404 returned error can't find the container with id a163cf5d8017b29a74a14354aff78de079fbf080080ff1ed136b8e435671b1f9 Oct 30 15:31:46.666246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.666227 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw"] Oct 30 15:31:46.982428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.982343 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" event={"ID":"56ad4837-6451-4aab-9bef-a0eba0842edf","Type":"ContainerStarted","Data":"24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf"} Oct 30 15:31:46.982428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.982408 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" event={"ID":"56ad4837-6451-4aab-9bef-a0eba0842edf","Type":"ContainerStarted","Data":"a163cf5d8017b29a74a14354aff78de079fbf080080ff1ed136b8e435671b1f9"} Oct 30 15:31:46.982899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.982437 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:31:46.984120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.984094 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" event={"ID":"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f","Type":"ContainerStarted","Data":"cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca"} Oct 30 15:31:46.984238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.984138 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" event={"ID":"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f","Type":"ContainerStarted","Data":"87b9cbe6c6d290f09dbe74a2f01543b4b456b72fb9dceef43a9fa11bf5df5cbb"} Oct 30 15:31:46.984287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:46.984230 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:47.043419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:47.043376 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" podStartSLOduration=2.043361223 podStartE2EDuration="2.043361223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:47.039606887 +0000 UTC m=+2610.998648292" watchObservedRunningTime="2026-10-30 15:31:47.043361223 +0000 UTC m=+2611.002402619" Oct 30 15:31:47.095457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:47.095416 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" podStartSLOduration=2.095404207 podStartE2EDuration="2.095404207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:47.093055561 +0000 UTC m=+2611.052096968" watchObservedRunningTime="2026-10-30 15:31:47.095404207 +0000 UTC m=+2611.054445612" Oct 30 15:31:50.769514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.769473 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v"] Oct 30 15:31:50.774783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.774764 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:50.798297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.798269 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v"] Oct 30 15:31:50.807234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.807207 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-648b1c\"" Oct 30 15:31:50.812428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.812400 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn"] Oct 30 15:31:50.817141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.817116 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:50.837872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.837844 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-648b1c-dockercfg-lmrds\"" Oct 30 15:31:50.843929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.843907 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn"] Oct 30 15:31:50.867285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.867258 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:50.867431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.867343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwpx5\" (UniqueName: \"kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:50.867503 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.867436 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz9m\" (UniqueName: \"kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m\") pod \"payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:50.867560 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.867543 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:50.986377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.974458 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:50.986377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.974562 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwpx5\" (UniqueName: \"kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:50.986377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.974599 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz9m\" (UniqueName: \"kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m\") pod \"payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:50.986377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.974713 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:50.986377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.975834 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:50.990372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:50.988747 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:51.063412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:51.051165 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz9m\" (UniqueName: \"kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m\") pod \"payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:51.063412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:51.060298 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwpx5\" (UniqueName: \"kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5\") pod \"payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:51.092215 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:51.090989 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:51.133748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:51.133715 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:51.471846 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:31:51.471810 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6449fff6_d95c_414e_9f61_f101f71d81d6.slice/crio-2f5fc3eb7a756f4ee4fa01fed5561ebca8c13ae3ea0d097a70f45ae7cb62ea40 WatchSource:0}: Error finding container 2f5fc3eb7a756f4ee4fa01fed5561ebca8c13ae3ea0d097a70f45ae7cb62ea40: Status 404 returned error can't find the container with id 2f5fc3eb7a756f4ee4fa01fed5561ebca8c13ae3ea0d097a70f45ae7cb62ea40 Oct 30 15:31:51.472858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:51.472831 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v"] Oct 30 15:31:51.534187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:51.534161 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn"] Oct 30 15:31:52.095379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.095329 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" event={"ID":"6449fff6-d95c-414e-9f61-f101f71d81d6","Type":"ContainerStarted","Data":"a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d"} Oct 30 15:31:52.095857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.095433 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" event={"ID":"6449fff6-d95c-414e-9f61-f101f71d81d6","Type":"ContainerStarted","Data":"2f5fc3eb7a756f4ee4fa01fed5561ebca8c13ae3ea0d097a70f45ae7cb62ea40"} Oct 30 15:31:52.095857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.095469 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:31:52.097078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.097052 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" event={"ID":"aee67f93-6f2b-4831-ab96-964454b06b13","Type":"ContainerStarted","Data":"e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7"} Oct 30 15:31:52.097207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.097094 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" event={"ID":"aee67f93-6f2b-4831-ab96-964454b06b13","Type":"ContainerStarted","Data":"d7d4f331788128f9b3d568028524e2aadd821834ac4cdc4437a0da60b0860ec3"} Oct 30 15:31:52.097207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.097114 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:31:52.140596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.140538 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" podStartSLOduration=2.140521735 podStartE2EDuration="2.140521735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:52.138938948 +0000 UTC m=+2616.097980354" watchObservedRunningTime="2026-10-30 15:31:52.140521735 +0000 UTC m=+2616.099563140" Oct 30 15:31:52.187759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:52.187713 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" podStartSLOduration=2.187700436 podStartE2EDuration="2.187700436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:31:52.184285158 +0000 UTC m=+2616.143326563" watchObservedRunningTime="2026-10-30 15:31:52.187700436 +0000 UTC m=+2616.146741840" Oct 30 15:31:54.895973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:54.895944 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:31:54.896461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:54.896426 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:31:58.006771 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:58.006214 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:31:58.006771 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:31:58.006341 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:32:03.107133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:03.107103 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:32:03.107538 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:03.107165 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:32:09.204702 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:32:09.204666 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:32:25.228379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:25.228331 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7"] Oct 30 15:32:25.266866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:25.266831 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v"] Oct 30 15:32:30.228771 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.228716 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" podUID="7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" containerName="payload-processing" containerID="cri-o://b2244c624d08145417d2f047a924df5e8e815156c68ce287b6bbfc5f05d985aa" gracePeriod=25 Oct 30 15:32:30.268500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.268459 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" podUID="e07f6acc-8ab7-4c60-96ce-1281f584a867" containerName="payload-pre-processing" containerID="cri-o://8de0af4f85ff6152240a96a9885f73e7c39a58300e2edf58256fb63278030d60" gracePeriod=25 Oct 30 15:32:30.397480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.397451 6554 generic.go:356] "Generic (PLEG): container finished" podID="e07f6acc-8ab7-4c60-96ce-1281f584a867" containerID="8de0af4f85ff6152240a96a9885f73e7c39a58300e2edf58256fb63278030d60" exitCode=0 Oct 30 15:32:30.397621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.397521 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" event={"ID":"e07f6acc-8ab7-4c60-96ce-1281f584a867","Type":"ContainerDied","Data":"8de0af4f85ff6152240a96a9885f73e7c39a58300e2edf58256fb63278030d60"} Oct 30 15:32:30.401645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.401611 6554 generic.go:356] "Generic (PLEG): container finished" podID="7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" containerID="b2244c624d08145417d2f047a924df5e8e815156c68ce287b6bbfc5f05d985aa" exitCode=0 Oct 30 15:32:30.401780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.401689 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" event={"ID":"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1","Type":"ContainerDied","Data":"b2244c624d08145417d2f047a924df5e8e815156c68ce287b6bbfc5f05d985aa"} Oct 30 15:32:30.609703 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.604889 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:32:30.643732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.643707 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:32:30.685281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.684580 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw\" (UniqueName: \"kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw\") pod \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " Oct 30 15:32:30.685281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.684800 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume\") pod \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\" (UID: \"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1\") " Oct 30 15:32:30.685281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.684926 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume\") pod \"e07f6acc-8ab7-4c60-96ce-1281f584a867\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " Oct 30 15:32:30.685281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.684961 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr\" (UniqueName: \"kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr\") pod \"e07f6acc-8ab7-4c60-96ce-1281f584a867\" (UID: \"e07f6acc-8ab7-4c60-96ce-1281f584a867\") " Oct 30 15:32:30.685626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.685475 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume" pod "e07f6acc-8ab7-4c60-96ce-1281f584a867" (UID: "e07f6acc-8ab7-4c60-96ce-1281f584a867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:32:30.685734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.685709 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07f6acc-8ab7-4c60-96ce-1281f584a867-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:32:30.686666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.686164 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume" pod "7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" (UID: "7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:32:30.688301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.688268 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr" pod "e07f6acc-8ab7-4c60-96ce-1281f584a867" (UID: "e07f6acc-8ab7-4c60-96ce-1281f584a867"). InnerVolumeSpecName "kube-api-access-rbgdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:30.691648 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.691620 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw" pod "7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" (UID: "7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1"). InnerVolumeSpecName "kube-api-access-t47cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:32:30.787824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.787723 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:32:30.787824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.787783 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbgdr\" (UniqueName: \"kubernetes.io/projected/e07f6acc-8ab7-4c60-96ce-1281f584a867-kube-api-access-rbgdr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:32:30.787824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:30.787798 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t47cw\" (UniqueName: \"kubernetes.io/projected/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1-kube-api-access-t47cw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:32:31.410641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.410615 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" Oct 30 15:32:31.411066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.410650 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7" event={"ID":"7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1","Type":"ContainerDied","Data":"228046e8f2a5ecc87530981c17a6df8928917f5f652dd747b2657740e8b12897"} Oct 30 15:32:31.411066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.410708 6554 scope.go:122] "RemoveContainer" containerID="b2244c624d08145417d2f047a924df5e8e815156c68ce287b6bbfc5f05d985aa" Oct 30 15:32:31.412254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.412232 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" event={"ID":"e07f6acc-8ab7-4c60-96ce-1281f584a867","Type":"ContainerDied","Data":"5d5d8817a9bcae22ee2f309d07307fe4985694365e11b0b3ef80e2c6a0abecbc"} Oct 30 15:32:31.412378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.412237 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v" Oct 30 15:32:31.423981 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.423962 6554 scope.go:122] "RemoveContainer" containerID="8de0af4f85ff6152240a96a9885f73e7c39a58300e2edf58256fb63278030d60" Oct 30 15:32:31.484822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.484797 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7"] Oct 30 15:32:31.490068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.490043 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e5423a-554d9c8b6b-dvkm7"] Oct 30 15:32:31.566422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.562397 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v"] Oct 30 15:32:31.581539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:31.581507 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e5423a-65665f9879-ts57v"] Oct 30 15:32:32.157115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:32.157082 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" path="/var/lib/kubelet/pods/7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1/volumes" Oct 30 15:32:32.157591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:32.157576 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07f6acc-8ab7-4c60-96ce-1281f584a867" path="/var/lib/kubelet/pods/e07f6acc-8ab7-4c60-96ce-1281f584a867/volumes" Oct 30 15:32:39.211010 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:32:39.210966 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:32:44.644069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.644036 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p"] Oct 30 15:32:44.644878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.644861 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e07f6acc-8ab7-4c60-96ce-1281f584a867" containerName="payload-pre-processing" Oct 30 15:32:44.644926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.644880 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07f6acc-8ab7-4c60-96ce-1281f584a867" containerName="payload-pre-processing" Oct 30 15:32:44.644926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.644906 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" containerName="payload-processing" Oct 30 15:32:44.644926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.644912 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" containerName="payload-processing" Oct 30 15:32:44.645058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.645047 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07f6acc-8ab7-4c60-96ce-1281f584a867" containerName="payload-pre-processing" Oct 30 15:32:44.645095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.645065 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a46e4e0-8e8e-4dc6-a08c-7bdae7b6cca1" containerName="payload-processing" Oct 30 15:32:44.648994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.648974 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.678835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.678814 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-648b1c-serving-cert\"" Oct 30 15:32:44.684095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.684067 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p"] Oct 30 15:32:44.686003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.685973 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.686134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.686109 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.686203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.686177 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.686512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.686485 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqnr4\" (UniqueName: \"kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.788467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.788433 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqnr4\" (UniqueName: \"kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.788651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.788569 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.788651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.788629 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.788782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.788669 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.788847 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:32:44.788783 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-serving-cert: secret "maas-api-e2e-ipp-b-648b1c-serving-cert" not found Oct 30 15:32:44.788913 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:32:44.788855 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls podName:da273e05-66b8-4ce7-8a5a-8a0ba247c554 nodeName:}" failed. No retries permitted until 2026-10-30 15:32:45.288836556 +0000 UTC m=+2669.247877952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls") pod "maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" (UID: "da273e05-66b8-4ce7-8a5a-8a0ba247c554") : secret "maas-api-e2e-ipp-b-648b1c-serving-cert" not found Oct 30 15:32:44.789423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.789344 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:44.791472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:44.791447 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:45.014431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:45.005580 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqnr4\" (UniqueName: \"kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:45.363432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:45.362693 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:45.400579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:45.400543 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\") pod \"maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:45.568463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:45.564633 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:46.241600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:46.241559 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p"] Oct 30 15:32:46.242052 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:32:46.241919 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda273e05_66b8_4ce7_8a5a_8a0ba247c554.slice/crio-912de47a50989213734f01d65861cbbbf258cfa325839c752a644148ce5774ef WatchSource:0}: Error finding container 912de47a50989213734f01d65861cbbbf258cfa325839c752a644148ce5774ef: Status 404 returned error can't find the container with id 912de47a50989213734f01d65861cbbbf258cfa325839c752a644148ce5774ef Oct 30 15:32:46.243553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:46.243534 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:32:46.606940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:46.606903 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" event={"ID":"da273e05-66b8-4ce7-8a5a-8a0ba247c554","Type":"ContainerStarted","Data":"912de47a50989213734f01d65861cbbbf258cfa325839c752a644148ce5774ef"} Oct 30 15:32:47.615602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:47.615565 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" event={"ID":"da273e05-66b8-4ce7-8a5a-8a0ba247c554","Type":"ContainerStarted","Data":"81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa"} Oct 30 15:32:47.616115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:47.615658 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:47.666711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:47.666665 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" podStartSLOduration=3.256057032 podStartE2EDuration="3.666652551s" totalImagesPullingTime="410.595519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:47.66329087 +0000 UTC m=+2671.622332316" watchObservedRunningTime="2026-10-30 15:32:47.666652551 +0000 UTC m=+2671.625693956" Oct 30 15:32:53.629359 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:53.629316 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:32:57.314446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.314411 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh"] Oct 30 15:32:57.319389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.319371 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.338054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.338024 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh"] Oct 30 15:32:57.347614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.347593 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5e8bf0\"" Oct 30 15:32:57.347762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.347638 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5e8bf0-dockercfg-8thlr\"" Oct 30 15:32:57.408373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.406139 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.408373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.406221 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrdr9\" (UniqueName: \"kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.541370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.511394 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.541370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.511479 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrdr9\" (UniqueName: \"kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.541370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.513037 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.600280 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.583933 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrdr9\" (UniqueName: \"kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9\") pod \"payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.636395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.632765 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9"] Oct 30 15:32:57.637104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.636848 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:57.642619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.642596 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9"] Oct 30 15:32:57.642855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.642838 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.714493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.714462 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.714856 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.714832 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qs7\" (UniqueName: \"kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7\") pod \"payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.816779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.816744 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.816976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.816876 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qs7\" (UniqueName: \"kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7\") pod \"payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.817554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.817529 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.849998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.849773 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qs7\" (UniqueName: \"kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7\") pod \"payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:57.988160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:57.988131 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh"] Oct 30 15:32:57.988393 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:32:57.988364 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b242f8e_7c76_479b_8511_cb2ed005599a.slice/crio-9b761bfb58a98bfc8c80b929bdb7965b6bc39f1e10932251107f30b8fab8e10c WatchSource:0}: Error finding container 9b761bfb58a98bfc8c80b929bdb7965b6bc39f1e10932251107f30b8fab8e10c: Status 404 returned error can't find the container with id 9b761bfb58a98bfc8c80b929bdb7965b6bc39f1e10932251107f30b8fab8e10c Oct 30 15:32:58.005987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.005965 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:58.370573 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:32:58.370530 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c19d41_8971_42d6_9406_2fef7d2cec43.slice/crio-cd5f5ff8b610347c0a5ac521fb912d768afc95a05348e945ad0641a8a93865c4 WatchSource:0}: Error finding container cd5f5ff8b610347c0a5ac521fb912d768afc95a05348e945ad0641a8a93865c4: Status 404 returned error can't find the container with id cd5f5ff8b610347c0a5ac521fb912d768afc95a05348e945ad0641a8a93865c4 Oct 30 15:32:58.374144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.373982 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9"] Oct 30 15:32:58.721261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.721227 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" event={"ID":"7b242f8e-7c76-479b-8511-cb2ed005599a","Type":"ContainerStarted","Data":"3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313"} Oct 30 15:32:58.721478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.721295 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:32:58.721478 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.721321 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" event={"ID":"7b242f8e-7c76-479b-8511-cb2ed005599a","Type":"ContainerStarted","Data":"9b761bfb58a98bfc8c80b929bdb7965b6bc39f1e10932251107f30b8fab8e10c"} Oct 30 15:32:58.723065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.723032 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" event={"ID":"f3c19d41-8971-42d6-9406-2fef7d2cec43","Type":"ContainerStarted","Data":"6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66"} Oct 30 15:32:58.723195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.723072 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" event={"ID":"f3c19d41-8971-42d6-9406-2fef7d2cec43","Type":"ContainerStarted","Data":"cd5f5ff8b610347c0a5ac521fb912d768afc95a05348e945ad0641a8a93865c4"} Oct 30 15:32:58.723195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.723098 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:32:58.766544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.766504 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" podStartSLOduration=1.766493345 podStartE2EDuration="1.766493345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:58.764187395 +0000 UTC m=+2682.723228821" watchObservedRunningTime="2026-10-30 15:32:58.766493345 +0000 UTC m=+2682.725534749" Oct 30 15:32:58.813925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:32:58.813882 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" podStartSLOduration=1.813870932 podStartE2EDuration="1.813870932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:32:58.811538044 +0000 UTC m=+2682.770579450" watchObservedRunningTime="2026-10-30 15:32:58.813870932 +0000 UTC m=+2682.772912377" Oct 30 15:33:09.210667 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:09.210629 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:33:09.734697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:09.734671 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:33:09.734897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:09.734743 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:33:10.157159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.157127 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf"] Oct 30 15:33:10.162485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.162464 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.181775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.181750 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-bcd8d6-openshift-default-dockercfg-vhgnw\"" Oct 30 15:33:10.188695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.188669 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf"] Oct 30 15:33:10.277057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277026 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg2zd\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277097 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277157 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277216 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277306 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277470 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277524 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277604 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.277990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.277792 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379467 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379534 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379594 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379746 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379814 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379840 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379927 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379952 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.379998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.379970 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg2zd\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.380155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.380018 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.380155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.380065 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.380489 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.380463 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.380623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.380603 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.380697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.380663 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.382277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.382256 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.382476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.382454 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.404828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.404801 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.405048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.405032 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg2zd\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd\") pod \"e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.479242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.479157 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:10.811674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.811642 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf"] Oct 30 15:33:10.811886 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:33:10.811863 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146b4be7_602b_4930_9bed_d94446ea35b4.slice/crio-aa778791df8fa45169a72dfffd9e99cc37b5c1f95620bcfaf4e9fc9098d14da5 WatchSource:0}: Error finding container aa778791df8fa45169a72dfffd9e99cc37b5c1f95620bcfaf4e9fc9098d14da5: Status 404 returned error can't find the container with id aa778791df8fa45169a72dfffd9e99cc37b5c1f95620bcfaf4e9fc9098d14da5 Oct 30 15:33:10.814697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.814662 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:10.814783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.814741 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:10.814783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.814771 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:10.836383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:10.836331 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" event={"ID":"146b4be7-602b-4930-9bed-d94446ea35b4","Type":"ContainerStarted","Data":"aa778791df8fa45169a72dfffd9e99cc37b5c1f95620bcfaf4e9fc9098d14da5"} Oct 30 15:33:11.845150 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:11.845116 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" event={"ID":"146b4be7-602b-4930-9bed-d94446ea35b4","Type":"ContainerStarted","Data":"d6b3a26c15bc77fdd98108f45b6fc97892f598d9251dece8099422e56fcc2160"} Oct 30 15:33:11.911819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:11.911769 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podStartSLOduration=1.911754484 podStartE2EDuration="1.911754484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:11.908612048 +0000 UTC m=+2695.867653452" watchObservedRunningTime="2026-10-30 15:33:11.911754484 +0000 UTC m=+2695.870795888" Oct 30 15:33:12.479441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:12.479405 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:12.481013 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:12.480987 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 15:33:12.481124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:12.481037 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 15:33:13.480418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:13.480383 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 15:33:13.480919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:13.480432 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 15:33:14.479757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:14.479728 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 15:33:14.479955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:14.479786 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 15:33:15.479802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:15.479764 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 15:33:15.480200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:15.479827 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 15:33:16.484782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:16.484747 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" start-of-body= Oct 30 15:33:16.485435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:16.485405 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.221:15021/healthz/ready\": dial tcp 10.134.0.221:15021: connect: connection refused" Oct 30 15:33:17.485527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:17.485487 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:17.486143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:17.486122 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:17.487036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:17.487018 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:33:30.103457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:30.103345 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s"] Oct 30 15:33:30.104027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:30.103622 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" containerID="cri-o://22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc" gracePeriod=30 Oct 30 15:33:35.432831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.432804 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:33:35.536814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.536784 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.536856 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.536924 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.536961 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.536986 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537081 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:35.537225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537109 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537225 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537160 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.537805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537776 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:35.537989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537808 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:35.537989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537832 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:35.538148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.538007 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:35.538148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.537433 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.538148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.538126 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token\") pod \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\" (UID: \"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2\") " Oct 30 15:33:35.539116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.539088 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.539221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.539119 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.539221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.539133 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.539221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.539145 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.539221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.539159 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.539917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.539894 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:33:35.540545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.540520 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:33:35.540662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.540602 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "kube-api-access-gzbpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:35.540884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.540858 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token" pod "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" (UID: "2ba1a5c4-6120-4e47-83ae-f23cd063b1f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:35.641369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.641173 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.641369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.641210 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.641369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.641225 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:35.641369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:35.641238 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbpz\" (UniqueName: \"kubernetes.io/projected/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2-kube-api-access-gzbpz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:36.187590 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.187551 6554 generic.go:356] "Generic (PLEG): container finished" podID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerID="22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc" exitCode=0 Oct 30 15:33:36.187775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.187599 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" Oct 30 15:33:36.187775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.187630 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" event={"ID":"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2","Type":"ContainerDied","Data":"22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc"} Oct 30 15:33:36.187775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.187702 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s" event={"ID":"2ba1a5c4-6120-4e47-83ae-f23cd063b1f2","Type":"ContainerDied","Data":"9c3b7ba46d456bbc7462140b6b996d9c9b94a35b69d52854189c8d1ee05267e6"} Oct 30 15:33:36.187775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.187728 6554 scope.go:122] "RemoveContainer" containerID="22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc" Oct 30 15:33:36.200162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.200146 6554 scope.go:122] "RemoveContainer" containerID="22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc" Oct 30 15:33:36.200447 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:36.200429 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc\": container with ID starting with 22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc not found: ID does not exist" containerID="22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc" Oct 30 15:33:36.200506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.200455 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc"} err="failed to get container status \"22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc\": rpc error: code = NotFound desc = could not find container \"22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc\": container with ID starting with 22db47e0106f3b86397fd12188b24c6386976ecb8a6b5a4eb33b7509999775fc not found: ID does not exist" Oct 30 15:33:36.249080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.249050 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s"] Oct 30 15:33:36.287811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:36.287775 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-rhv8s"] Oct 30 15:33:37.090944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.090910 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4"] Oct 30 15:33:37.091830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.091811 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" Oct 30 15:33:37.091830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.091830 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" Oct 30 15:33:37.092001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.091991 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" containerName="istio-proxy" Oct 30 15:33:37.097461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.097442 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.119256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.119232 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4"] Oct 30 15:33:37.127557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.127533 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-631d84-openshift-default-dockercfg-ns62r\"" Oct 30 15:33:37.192951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.192912 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.192961 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz8gw\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193137 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193180 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193293 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193318 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193379 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193470 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.193605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.193515 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295106 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295147 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295221 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295246 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295286 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295368 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295410 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295576 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295606 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz8gw\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.295778 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295761 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.296010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295979 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.296010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.295988 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.296165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.296033 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.296218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.296201 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.297684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.297660 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.298040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.298016 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.321359 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.321324 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.321666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.321642 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz8gw\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw\") pod \"e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.411517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.411488 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:37.783126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.783096 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4"] Oct 30 15:33:37.783420 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:33:37.783395 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde262476_c55d_4eb9_84dc_07a4eedbd1b8.slice/crio-b4643e9c79c8bedf9213a960687fbd7f7729485ddad66bd5f56abff26319a03f WatchSource:0}: Error finding container b4643e9c79c8bedf9213a960687fbd7f7729485ddad66bd5f56abff26319a03f: Status 404 returned error can't find the container with id b4643e9c79c8bedf9213a960687fbd7f7729485ddad66bd5f56abff26319a03f Oct 30 15:33:37.787010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.786971 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:37.787099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.787057 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:37.787146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:37.787099 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:38.183401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.177469 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba1a5c4-6120-4e47-83ae-f23cd063b1f2" path="/var/lib/kubelet/pods/2ba1a5c4-6120-4e47-83ae-f23cd063b1f2/volumes" Oct 30 15:33:38.207900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.207866 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" event={"ID":"de262476-c55d-4eb9-84dc-07a4eedbd1b8","Type":"ContainerStarted","Data":"48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e"} Oct 30 15:33:38.208054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.207918 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" event={"ID":"de262476-c55d-4eb9-84dc-07a4eedbd1b8","Type":"ContainerStarted","Data":"b4643e9c79c8bedf9213a960687fbd7f7729485ddad66bd5f56abff26319a03f"} Oct 30 15:33:38.288382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.283536 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podStartSLOduration=1.28351644 podStartE2EDuration="1.28351644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:33:38.267766196 +0000 UTC m=+2722.226807600" watchObservedRunningTime="2026-10-30 15:33:38.28351644 +0000 UTC m=+2722.242557848" Oct 30 15:33:38.412413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.412380 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:38.413667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.413641 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 15:33:38.413824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:38.413686 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 15:33:39.113377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:39.113224 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv"] Oct 30 15:33:39.133893 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:39.133857 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6"] Oct 30 15:33:39.248898 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:39.248043 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:33:39.413014 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:39.412977 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 15:33:39.413313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:39.413284 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 15:33:40.412677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:40.412648 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 15:33:40.413099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:40.412693 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 15:33:41.412265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:41.412225 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 15:33:41.412495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:41.412308 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 15:33:42.412188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:42.412154 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 15:33:42.412728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:42.412206 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 15:33:42.832527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:42.832438 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:33:42.832527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:42.832488 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:33:43.412085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:43.412050 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" start-of-body= Oct 30 15:33:43.412264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:43.412115 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.222:15021/healthz/ready\": dial tcp 10.134.0.222:15021: connect: connection refused" Oct 30 15:33:44.114102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.114054 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" podUID="685fde02-9c86-4a8d-8f97-fa0981a18202" containerName="payload-processing" containerID="cri-o://df8d415be5b5b504e38664c7ba2aaca94120eb5c85d88c573ce6e969f461bbc5" gracePeriod=25 Oct 30 15:33:44.134386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.134339 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" podUID="ab596006-3424-46d3-8e0d-a12e11e95d1d" containerName="payload-pre-processing" containerID="cri-o://f643782a9eed318bfadda67ceada0e335651744db692015d0d5cec8c304c2536" gracePeriod=25 Oct 30 15:33:44.318106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.318070 6554 generic.go:356] "Generic (PLEG): container finished" podID="ab596006-3424-46d3-8e0d-a12e11e95d1d" containerID="f643782a9eed318bfadda67ceada0e335651744db692015d0d5cec8c304c2536" exitCode=0 Oct 30 15:33:44.318293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.318145 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" event={"ID":"ab596006-3424-46d3-8e0d-a12e11e95d1d","Type":"ContainerDied","Data":"f643782a9eed318bfadda67ceada0e335651744db692015d0d5cec8c304c2536"} Oct 30 15:33:44.320118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.320096 6554 generic.go:356] "Generic (PLEG): container finished" podID="685fde02-9c86-4a8d-8f97-fa0981a18202" containerID="df8d415be5b5b504e38664c7ba2aaca94120eb5c85d88c573ce6e969f461bbc5" exitCode=0 Oct 30 15:33:44.320235 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.320178 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" event={"ID":"685fde02-9c86-4a8d-8f97-fa0981a18202","Type":"ContainerDied","Data":"df8d415be5b5b504e38664c7ba2aaca94120eb5c85d88c573ce6e969f461bbc5"} Oct 30 15:33:44.738021 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.737042 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:44.740658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.738224 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:44.740994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.740966 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:33:44.792496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.792467 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:33:44.838508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.838483 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume\") pod \"685fde02-9c86-4a8d-8f97-fa0981a18202\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " Oct 30 15:33:44.838670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.838559 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg\" (UniqueName: \"kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg\") pod \"685fde02-9c86-4a8d-8f97-fa0981a18202\" (UID: \"685fde02-9c86-4a8d-8f97-fa0981a18202\") " Oct 30 15:33:44.838842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.838819 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume" pod "685fde02-9c86-4a8d-8f97-fa0981a18202" (UID: "685fde02-9c86-4a8d-8f97-fa0981a18202"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:44.839427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.839406 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/685fde02-9c86-4a8d-8f97-fa0981a18202-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:44.840925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.840905 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg" pod "685fde02-9c86-4a8d-8f97-fa0981a18202" (UID: "685fde02-9c86-4a8d-8f97-fa0981a18202"). InnerVolumeSpecName "kube-api-access-26njg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:44.862125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.862104 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:33:44.940502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.940426 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7\" (UniqueName: \"kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7\") pod \"ab596006-3424-46d3-8e0d-a12e11e95d1d\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " Oct 30 15:33:44.940668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.940563 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume\") pod \"ab596006-3424-46d3-8e0d-a12e11e95d1d\" (UID: \"ab596006-3424-46d3-8e0d-a12e11e95d1d\") " Oct 30 15:33:44.941243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.941214 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume" pod "ab596006-3424-46d3-8e0d-a12e11e95d1d" (UID: "ab596006-3424-46d3-8e0d-a12e11e95d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:44.941641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.941398 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26njg\" (UniqueName: \"kubernetes.io/projected/685fde02-9c86-4a8d-8f97-fa0981a18202-kube-api-access-26njg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:44.941641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.941421 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab596006-3424-46d3-8e0d-a12e11e95d1d-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:44.944566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:44.944544 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7" pod "ab596006-3424-46d3-8e0d-a12e11e95d1d" (UID: "ab596006-3424-46d3-8e0d-a12e11e95d1d"). InnerVolumeSpecName "kube-api-access-rs6k7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:45.043133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.043099 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs6k7\" (UniqueName: \"kubernetes.io/projected/ab596006-3424-46d3-8e0d-a12e11e95d1d-kube-api-access-rs6k7\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:45.329272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.329242 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" Oct 30 15:33:45.329466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.329243 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6" event={"ID":"ab596006-3424-46d3-8e0d-a12e11e95d1d","Type":"ContainerDied","Data":"af6c313f165fde5db7a9283c3c89e778989cd982fc5cddc9f61858edc8b74c99"} Oct 30 15:33:45.329466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.329393 6554 scope.go:122] "RemoveContainer" containerID="f643782a9eed318bfadda67ceada0e335651744db692015d0d5cec8c304c2536" Oct 30 15:33:45.330953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.330934 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" Oct 30 15:33:45.331029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.330961 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv" event={"ID":"685fde02-9c86-4a8d-8f97-fa0981a18202","Type":"ContainerDied","Data":"9523f35b364932b188676f009ca8c77a9930011673758b0ed4cecd60e769863e"} Oct 30 15:33:45.342826 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.342808 6554 scope.go:122] "RemoveContainer" containerID="df8d415be5b5b504e38664c7ba2aaca94120eb5c85d88c573ce6e969f461bbc5" Oct 30 15:33:45.411849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.387845 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6"] Oct 30 15:33:45.411849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.394035 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-631d84-796b6dfc9f-86rq6"] Oct 30 15:33:45.607837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.431915 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv"] Oct 30 15:33:45.607837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:45.435368 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-631d84-7857b77db6-mzbmv"] Oct 30 15:33:46.157044 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.157013 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685fde02-9c86-4a8d-8f97-fa0981a18202" path="/var/lib/kubelet/pods/685fde02-9c86-4a8d-8f97-fa0981a18202/volumes" Oct 30 15:33:46.157494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.157479 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab596006-3424-46d3-8e0d-a12e11e95d1d" path="/var/lib/kubelet/pods/ab596006-3424-46d3-8e0d-a12e11e95d1d/volumes" Oct 30 15:33:46.342505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.342475 6554 generic.go:356] "Generic (PLEG): container finished" podID="01f70536-e123-4dad-810c-b77ab92b8c04" containerID="ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f" exitCode=0 Oct 30 15:33:46.342690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.342541 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" event={"ID":"01f70536-e123-4dad-810c-b77ab92b8c04","Type":"ContainerDied","Data":"ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f"} Oct 30 15:33:46.343120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.343099 6554 scope.go:122] "RemoveContainer" containerID="ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f" Oct 30 15:33:46.344881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.344853 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:46.344968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.344922 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:46.344968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.344953 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:33:46.776806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.776732 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw"] Oct 30 15:33:46.812045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:46.812013 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486"] Oct 30 15:33:47.379681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:47.379649 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" event={"ID":"01f70536-e123-4dad-810c-b77ab92b8c04","Type":"ContainerStarted","Data":"8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a"} Oct 30 15:33:47.710819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:47.710785 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:33:47.711205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:47.711177 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:33:47.711419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:47.711385 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:33:48.713371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:48.710685 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:33:48.713371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:48.710746 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:33:49.707433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:49.707398 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:33:49.707609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:49.707456 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:33:50.707075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:50.707040 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:33:50.707552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:50.707105 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:33:51.707393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:51.707329 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:33:51.707818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:51.707410 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:33:51.778512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:51.778445 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" podUID="56ad4837-6451-4aab-9bef-a0eba0842edf" containerName="payload-processing" containerID="cri-o://24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf" gracePeriod=25 Oct 30 15:33:51.812344 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:51.812313 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" podUID="bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" containerName="payload-pre-processing" containerID="cri-o://cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca" gracePeriod=25 Oct 30 15:33:52.226803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.226772 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:33:52.233138 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.233116 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:33:52.324798 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.324705 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume\") pod \"56ad4837-6451-4aab-9bef-a0eba0842edf\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " Oct 30 15:33:52.324798 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.324776 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk\" (UniqueName: \"kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk\") pod \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " Oct 30 15:33:52.325374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.324930 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume\") pod \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\" (UID: \"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f\") " Oct 30 15:33:52.325374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.324964 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5\" (UniqueName: \"kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5\") pod \"56ad4837-6451-4aab-9bef-a0eba0842edf\" (UID: \"56ad4837-6451-4aab-9bef-a0eba0842edf\") " Oct 30 15:33:52.325945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.325908 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume" pod "bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" (UID: "bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:52.326572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.326252 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume" pod "56ad4837-6451-4aab-9bef-a0eba0842edf" (UID: "56ad4837-6451-4aab-9bef-a0eba0842edf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:52.329448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.328745 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk" pod "bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" (UID: "bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f"). InnerVolumeSpecName "kube-api-access-2hshk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:52.329448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.328940 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5" pod "56ad4837-6451-4aab-9bef-a0eba0842edf" (UID: "56ad4837-6451-4aab-9bef-a0eba0842edf"). InnerVolumeSpecName "kube-api-access-g22x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:52.421910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.421876 6554 generic.go:356] "Generic (PLEG): container finished" podID="56ad4837-6451-4aab-9bef-a0eba0842edf" containerID="24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf" exitCode=0 Oct 30 15:33:52.422087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.421925 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" Oct 30 15:33:52.422087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.421968 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" event={"ID":"56ad4837-6451-4aab-9bef-a0eba0842edf","Type":"ContainerDied","Data":"24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf"} Oct 30 15:33:52.422087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.422026 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw" event={"ID":"56ad4837-6451-4aab-9bef-a0eba0842edf","Type":"ContainerDied","Data":"a163cf5d8017b29a74a14354aff78de079fbf080080ff1ed136b8e435671b1f9"} Oct 30 15:33:52.422087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.422057 6554 scope.go:122] "RemoveContainer" containerID="24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf" Oct 30 15:33:52.423469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.423448 6554 generic.go:356] "Generic (PLEG): container finished" podID="bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" containerID="cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca" exitCode=0 Oct 30 15:33:52.423599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.423478 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" event={"ID":"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f","Type":"ContainerDied","Data":"cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca"} Oct 30 15:33:52.423599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.423492 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" Oct 30 15:33:52.423599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.423505 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486" event={"ID":"bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f","Type":"ContainerDied","Data":"87b9cbe6c6d290f09dbe74a2f01543b4b456b72fb9dceef43a9fa11bf5df5cbb"} Oct 30 15:33:52.427043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.427018 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ad4837-6451-4aab-9bef-a0eba0842edf-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:52.427135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.427047 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hshk\" (UniqueName: \"kubernetes.io/projected/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-kube-api-access-2hshk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:52.427135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.427063 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:52.427135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.427080 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22x5\" (UniqueName: \"kubernetes.io/projected/56ad4837-6451-4aab-9bef-a0eba0842edf-kube-api-access-g22x5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:52.434843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.434828 6554 scope.go:122] "RemoveContainer" containerID="24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf" Oct 30 15:33:52.435095 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:52.435078 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf\": container with ID starting with 24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf not found: ID does not exist" containerID="24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf" Oct 30 15:33:52.435146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.435101 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf"} err="failed to get container status \"24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf\": rpc error: code = NotFound desc = could not find container \"24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf\": container with ID starting with 24172f55d9e0f7066961776e369db41cabf8260226500450638fc01621fddbbf not found: ID does not exist" Oct 30 15:33:52.435146 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.435117 6554 scope.go:122] "RemoveContainer" containerID="cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca" Oct 30 15:33:52.445323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.445305 6554 scope.go:122] "RemoveContainer" containerID="cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca" Oct 30 15:33:52.445632 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:52.445609 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca\": container with ID starting with cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca not found: ID does not exist" containerID="cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca" Oct 30 15:33:52.445728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.445637 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca"} err="failed to get container status \"cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca\": rpc error: code = NotFound desc = could not find container \"cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca\": container with ID starting with cda2602046820a0159e66e04b429202a55f08c8171bed9aafd52f0e49e86e0ca not found: ID does not exist" Oct 30 15:33:52.512890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.512787 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw"] Oct 30 15:33:52.518272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.518243 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78b513-6cc74bff5b-skrtw"] Oct 30 15:33:52.594534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.594452 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486"] Oct 30 15:33:52.603946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.603887 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-78b513-88b5bddc5-24486"] Oct 30 15:33:52.707323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.707292 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:33:52.707508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.707343 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:33:52.721707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.721683 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v"] Oct 30 15:33:52.795196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:52.795162 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn"] Oct 30 15:33:53.817931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:53.817888 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:33:53.818734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:53.818711 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:33:53.819677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:53.819653 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:33:54.196187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:54.196156 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ad4837-6451-4aab-9bef-a0eba0842edf" path="/var/lib/kubelet/pods/56ad4837-6451-4aab-9bef-a0eba0842edf/volumes" Oct 30 15:33:54.196654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:54.196638 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" path="/var/lib/kubelet/pods/bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f/volumes" Oct 30 15:33:57.722333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:57.722286 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" podUID="6449fff6-d95c-414e-9f61-f101f71d81d6" containerName="payload-processing" containerID="cri-o://a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d" gracePeriod=25 Oct 30 15:33:57.796696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:57.796526 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" podUID="aee67f93-6f2b-4831-ab96-964454b06b13" containerName="payload-pre-processing" containerID="cri-o://e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7" gracePeriod=25 Oct 30 15:33:58.183604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.183581 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:33:58.188461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.188443 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:33:58.265901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.265870 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume\") pod \"6449fff6-d95c-414e-9f61-f101f71d81d6\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " Oct 30 15:33:58.266061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.265995 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5\" (UniqueName: \"kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5\") pod \"aee67f93-6f2b-4831-ab96-964454b06b13\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " Oct 30 15:33:58.266061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.266034 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume\") pod \"aee67f93-6f2b-4831-ab96-964454b06b13\" (UID: \"aee67f93-6f2b-4831-ab96-964454b06b13\") " Oct 30 15:33:58.266226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.266207 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m\" (UniqueName: \"kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m\") pod \"6449fff6-d95c-414e-9f61-f101f71d81d6\" (UID: \"6449fff6-d95c-414e-9f61-f101f71d81d6\") " Oct 30 15:33:58.266698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.266559 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume" pod "aee67f93-6f2b-4831-ab96-964454b06b13" (UID: "aee67f93-6f2b-4831-ab96-964454b06b13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:58.266698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.266643 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume" pod "6449fff6-d95c-414e-9f61-f101f71d81d6" (UID: "6449fff6-d95c-414e-9f61-f101f71d81d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:33:58.267166 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.267140 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6449fff6-d95c-414e-9f61-f101f71d81d6-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:58.267274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.267171 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aee67f93-6f2b-4831-ab96-964454b06b13-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:58.269403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.269328 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m" pod "6449fff6-d95c-414e-9f61-f101f71d81d6" (UID: "6449fff6-d95c-414e-9f61-f101f71d81d6"). InnerVolumeSpecName "kube-api-access-npz9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:58.273502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.273479 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5" pod "aee67f93-6f2b-4831-ab96-964454b06b13" (UID: "aee67f93-6f2b-4831-ab96-964454b06b13"). InnerVolumeSpecName "kube-api-access-cwpx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:33:58.369229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.369168 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwpx5\" (UniqueName: \"kubernetes.io/projected/aee67f93-6f2b-4831-ab96-964454b06b13-kube-api-access-cwpx5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:58.369229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.369193 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz9m\" (UniqueName: \"kubernetes.io/projected/6449fff6-d95c-414e-9f61-f101f71d81d6-kube-api-access-npz9m\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:33:58.630450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.630314 6554 generic.go:356] "Generic (PLEG): container finished" podID="6449fff6-d95c-414e-9f61-f101f71d81d6" containerID="a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d" exitCode=0 Oct 30 15:33:58.630450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.630424 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" Oct 30 15:33:58.630681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.630417 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" event={"ID":"6449fff6-d95c-414e-9f61-f101f71d81d6","Type":"ContainerDied","Data":"a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d"} Oct 30 15:33:58.630681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.630636 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v" event={"ID":"6449fff6-d95c-414e-9f61-f101f71d81d6","Type":"ContainerDied","Data":"2f5fc3eb7a756f4ee4fa01fed5561ebca8c13ae3ea0d097a70f45ae7cb62ea40"} Oct 30 15:33:58.630790 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.630691 6554 scope.go:122] "RemoveContainer" containerID="a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d" Oct 30 15:33:58.632521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.632496 6554 generic.go:356] "Generic (PLEG): container finished" podID="aee67f93-6f2b-4831-ab96-964454b06b13" containerID="e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7" exitCode=0 Oct 30 15:33:58.632658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.632553 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" Oct 30 15:33:58.632658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.632571 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" event={"ID":"aee67f93-6f2b-4831-ab96-964454b06b13","Type":"ContainerDied","Data":"e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7"} Oct 30 15:33:58.632658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.632630 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn" event={"ID":"aee67f93-6f2b-4831-ab96-964454b06b13","Type":"ContainerDied","Data":"d7d4f331788128f9b3d568028524e2aadd821834ac4cdc4437a0da60b0860ec3"} Oct 30 15:33:58.643586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.643567 6554 scope.go:122] "RemoveContainer" containerID="a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d" Oct 30 15:33:58.643900 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:58.643815 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d\": container with ID starting with a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d not found: ID does not exist" containerID="a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d" Oct 30 15:33:58.643900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.643841 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d"} err="failed to get container status \"a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d\": rpc error: code = NotFound desc = could not find container \"a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d\": container with ID starting with a1107a05683b0e7d1227b2b5956b3d398f5b49bc35e815a99eb6ca2cb007e42d not found: ID does not exist" Oct 30 15:33:58.643900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.643864 6554 scope.go:122] "RemoveContainer" containerID="e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7" Oct 30 15:33:58.654161 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.654142 6554 scope.go:122] "RemoveContainer" containerID="e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7" Oct 30 15:33:58.654420 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:33:58.654391 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7\": container with ID starting with e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7 not found: ID does not exist" containerID="e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7" Oct 30 15:33:58.654482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.654431 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7"} err="failed to get container status \"e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7\": rpc error: code = NotFound desc = could not find container \"e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7\": container with ID starting with e908731a81e816f4037737d243b6c3a34eee3e7ff9a31624635366b7e3cbeeb7 not found: ID does not exist" Oct 30 15:33:58.688510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.688486 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn"] Oct 30 15:33:58.701400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.701375 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-648b1c-5c64945b9f-wc4kn"] Oct 30 15:33:58.739068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.739047 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v"] Oct 30 15:33:58.746419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:33:58.746401 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-648b1c-688bd7bbb-2j68v"] Oct 30 15:34:00.156815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:00.156782 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6449fff6-d95c-414e-9f61-f101f71d81d6" path="/var/lib/kubelet/pods/6449fff6-d95c-414e-9f61-f101f71d81d6/volumes" Oct 30 15:34:00.157201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:00.157189 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aee67f93-6f2b-4831-ab96-964454b06b13" path="/var/lib/kubelet/pods/aee67f93-6f2b-4831-ab96-964454b06b13/volumes" Oct 30 15:34:01.463755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.463727 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq"] Oct 30 15:34:01.464514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464499 6554 cpu_manager.go:414] "containerMap: removing container" podUID="56ad4837-6451-4aab-9bef-a0eba0842edf" containerName="payload-processing" Oct 30 15:34:01.464514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464514 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ad4837-6451-4aab-9bef-a0eba0842edf" containerName="payload-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464531 6554 cpu_manager.go:414] "containerMap: removing container" podUID="bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" containerName="payload-pre-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464537 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" containerName="payload-pre-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464553 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ab596006-3424-46d3-8e0d-a12e11e95d1d" containerName="payload-pre-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464558 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab596006-3424-46d3-8e0d-a12e11e95d1d" containerName="payload-pre-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464569 6554 cpu_manager.go:414] "containerMap: removing container" podUID="aee67f93-6f2b-4831-ab96-964454b06b13" containerName="payload-pre-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464574 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee67f93-6f2b-4831-ab96-964454b06b13" containerName="payload-pre-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464582 6554 cpu_manager.go:414] "containerMap: removing container" podUID="685fde02-9c86-4a8d-8f97-fa0981a18202" containerName="payload-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464586 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685fde02-9c86-4a8d-8f97-fa0981a18202" containerName="payload-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464595 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6449fff6-d95c-414e-9f61-f101f71d81d6" containerName="payload-processing" Oct 30 15:34:01.464619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464600 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6449fff6-d95c-414e-9f61-f101f71d81d6" containerName="payload-processing" Oct 30 15:34:01.464930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464715 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ad4837-6451-4aab-9bef-a0eba0842edf" containerName="payload-processing" Oct 30 15:34:01.464930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464726 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6d4d91-ee9f-4c45-8fa6-48c4d97bb00f" containerName="payload-pre-processing" Oct 30 15:34:01.464930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464734 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab596006-3424-46d3-8e0d-a12e11e95d1d" containerName="payload-pre-processing" Oct 30 15:34:01.464930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464741 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee67f93-6f2b-4831-ab96-964454b06b13" containerName="payload-pre-processing" Oct 30 15:34:01.464930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464747 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="685fde02-9c86-4a8d-8f97-fa0981a18202" containerName="payload-processing" Oct 30 15:34:01.464930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.464754 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6449fff6-d95c-414e-9f61-f101f71d81d6" containerName="payload-processing" Oct 30 15:34:01.469966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.469951 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.488657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.488636 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq"] Oct 30 15:34:01.498042 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.498016 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-51bd24-serving-cert\"" Oct 30 15:34:01.530681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.530656 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.531005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.530983 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstp8\" (UniqueName: \"kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.531212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.531188 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.531294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.531273 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.632722 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.632688 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstp8\" (UniqueName: \"kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.632866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.632847 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.632944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.632923 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.633004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.632960 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.633004 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:34:01.632978 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-serving-cert: secret "maas-api-e2e-ipp-a-51bd24-serving-cert" not found Oct 30 15:34:01.633119 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:34:01.633029 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls podName:c1a333af-1683-4f75-860f-db7128c29fb0 nodeName:}" failed. No retries permitted until 2026-10-30 15:34:02.133015017 +0000 UTC m=+2746.092056435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls") pod "maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" (UID: "c1a333af-1683-4f75-860f-db7128c29fb0") : secret "maas-api-e2e-ipp-a-51bd24-serving-cert" not found Oct 30 15:34:01.633622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.633602 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.635327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.635304 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:01.736446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:01.730099 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstp8\" (UniqueName: \"kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:02.152459 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:02.152429 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:02.154965 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:02.154937 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:02.384819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:02.384790 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:02.703783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:02.703756 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq"] Oct 30 15:34:02.704483 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:34:02.704454 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a333af_1683_4f75_860f_db7128c29fb0.slice/crio-db14cfcdfb586e16e246702fae15b803451a678fd029ddf0ad052e8f84423163 WatchSource:0}: Error finding container db14cfcdfb586e16e246702fae15b803451a678fd029ddf0ad052e8f84423163: Status 404 returned error can't find the container with id db14cfcdfb586e16e246702fae15b803451a678fd029ddf0ad052e8f84423163 Oct 30 15:34:02.760425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:02.760394 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" event={"ID":"c1a333af-1683-4f75-860f-db7128c29fb0","Type":"ContainerStarted","Data":"db14cfcdfb586e16e246702fae15b803451a678fd029ddf0ad052e8f84423163"} Oct 30 15:34:03.769589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:03.769544 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" event={"ID":"c1a333af-1683-4f75-860f-db7128c29fb0","Type":"ContainerStarted","Data":"b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1"} Oct 30 15:34:03.770017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:03.769659 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:03.830417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:03.820586 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" podStartSLOduration=2.49898899 podStartE2EDuration="2.820567412s" totalImagesPullingTime="321.578422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:34:03.814289702 +0000 UTC m=+2747.773331107" watchObservedRunningTime="2026-10-30 15:34:03.820567412 +0000 UTC m=+2747.779608818" Oct 30 15:34:09.231876 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:34:09.231839 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:34:09.783425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:09.783399 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:34:19.299638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:19.299603 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl"] Oct 30 15:34:19.300046 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:19.299791 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" containerID="cri-o://45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c" gracePeriod=30 Oct 30 15:34:24.591726 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.591700 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:34:24.679978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.679947 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.679998 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680030 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680129 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680204 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680416 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:24.680466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680426 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680576 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680622 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680674 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs\") pod \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\" (UID: \"630789f8-ab60-4ae5-b3b6-5bb64fb62223\") " Oct 30 15:34:24.680948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.680922 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:24.681582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.681139 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:24.681582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.681155 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:24.681582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.681501 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.681582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.681524 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.681582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.681538 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.681582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.681560 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.682144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.682122 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:34:24.682868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.682830 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "kube-api-access-2vqb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:24.683267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.683246 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:34:24.683479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.683454 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:34:24.684416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.684391 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token" pod "630789f8-ab60-4ae5-b3b6-5bb64fb62223" (UID: "630789f8-ab60-4ae5-b3b6-5bb64fb62223"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:34:24.784089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.783686 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqb9\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-kube-api-access-2vqb9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.784089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.783724 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.784089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.783739 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.784089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.783754 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.784089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.783767 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/630789f8-ab60-4ae5-b3b6-5bb64fb62223-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:34:24.930378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.930321 6554 generic.go:356] "Generic (PLEG): container finished" podID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerID="45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c" exitCode=0 Oct 30 15:34:24.930378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.930374 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" Oct 30 15:34:24.930554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.930411 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" event={"ID":"630789f8-ab60-4ae5-b3b6-5bb64fb62223","Type":"ContainerDied","Data":"45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c"} Oct 30 15:34:24.930554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.930470 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl" event={"ID":"630789f8-ab60-4ae5-b3b6-5bb64fb62223","Type":"ContainerDied","Data":"ec0910f55666494477396343e14ac663ff24f2ff3c209c2a9e8beb22f448f655"} Oct 30 15:34:24.930554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.930493 6554 scope.go:122] "RemoveContainer" containerID="45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c" Oct 30 15:34:24.941798 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.941777 6554 scope.go:122] "RemoveContainer" containerID="45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c" Oct 30 15:34:24.942083 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:34:24.942063 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c\": container with ID starting with 45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c not found: ID does not exist" containerID="45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c" Oct 30 15:34:24.942157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.942089 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c"} err="failed to get container status \"45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c\": rpc error: code = NotFound desc = could not find container \"45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c\": container with ID starting with 45ba7b8b3bb7335e9171393aa9452976d7e07527a0fdc6893f17e08f3cd9552c not found: ID does not exist" Oct 30 15:34:24.993034 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:24.993001 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl"] Oct 30 15:34:25.006214 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:25.006184 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-57zrl"] Oct 30 15:34:26.160096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:26.160055 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" path="/var/lib/kubelet/pods/630789f8-ab60-4ae5-b3b6-5bb64fb62223/volumes" Oct 30 15:34:39.238500 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:34:39.238457 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:34:39.700477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:39.700446 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz"] Oct 30 15:34:39.735282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:39.735254 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw"] Oct 30 15:34:44.701229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:44.701172 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerName="payload-processing" containerID="cri-o://2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc" gracePeriod=25 Oct 30 15:34:44.736198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:44.736141 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" containerName="payload-pre-processing" containerID="cri-o://3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26" gracePeriod=25 Oct 30 15:34:54.030989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:54.030945 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:34:54.031468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:54.031004 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:34:57.210730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:57.210652 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9"] Oct 30 15:34:57.245074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:34:57.245042 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh"] Oct 30 15:35:02.211885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.211848 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" podUID="f3c19d41-8971-42d6-9406-2fef7d2cec43" containerName="payload-processing" containerID="cri-o://6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66" gracePeriod=25 Oct 30 15:35:02.245761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.245723 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" podUID="7b242f8e-7c76-479b-8511-cb2ed005599a" containerName="payload-pre-processing" containerID="cri-o://3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313" gracePeriod=25 Oct 30 15:35:02.735137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.735109 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:35:02.779787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.779762 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:35:02.834293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.834200 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume\") pod \"f3c19d41-8971-42d6-9406-2fef7d2cec43\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " Oct 30 15:35:02.834293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.834280 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7\" (UniqueName: \"kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7\") pod \"f3c19d41-8971-42d6-9406-2fef7d2cec43\" (UID: \"f3c19d41-8971-42d6-9406-2fef7d2cec43\") " Oct 30 15:35:02.835982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.835950 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume" pod "f3c19d41-8971-42d6-9406-2fef7d2cec43" (UID: "f3c19d41-8971-42d6-9406-2fef7d2cec43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:02.837443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.837413 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7" pod "f3c19d41-8971-42d6-9406-2fef7d2cec43" (UID: "f3c19d41-8971-42d6-9406-2fef7d2cec43"). InnerVolumeSpecName "kube-api-access-w2qs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:02.936173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.936146 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume\") pod \"7b242f8e-7c76-479b-8511-cb2ed005599a\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " Oct 30 15:35:02.936329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.936304 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9\" (UniqueName: \"kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9\") pod \"7b242f8e-7c76-479b-8511-cb2ed005599a\" (UID: \"7b242f8e-7c76-479b-8511-cb2ed005599a\") " Oct 30 15:35:02.936583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.936560 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume" pod "7b242f8e-7c76-479b-8511-cb2ed005599a" (UID: "7b242f8e-7c76-479b-8511-cb2ed005599a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:02.936822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.936805 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b242f8e-7c76-479b-8511-cb2ed005599a-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:02.936887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.936828 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3c19d41-8971-42d6-9406-2fef7d2cec43-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:02.936887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.936842 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qs7\" (UniqueName: \"kubernetes.io/projected/f3c19d41-8971-42d6-9406-2fef7d2cec43-kube-api-access-w2qs7\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:02.938361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:02.938333 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9" pod "7b242f8e-7c76-479b-8511-cb2ed005599a" (UID: "7b242f8e-7c76-479b-8511-cb2ed005599a"). InnerVolumeSpecName "kube-api-access-zrdr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:03.048378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.044825 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrdr9\" (UniqueName: \"kubernetes.io/projected/7b242f8e-7c76-479b-8511-cb2ed005599a-kube-api-access-zrdr9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:03.289836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.289803 6554 generic.go:356] "Generic (PLEG): container finished" podID="7b242f8e-7c76-479b-8511-cb2ed005599a" containerID="3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313" exitCode=0 Oct 30 15:35:03.290283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.289846 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" Oct 30 15:35:03.290283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.289882 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" event={"ID":"7b242f8e-7c76-479b-8511-cb2ed005599a","Type":"ContainerDied","Data":"3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313"} Oct 30 15:35:03.290283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.289942 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh" event={"ID":"7b242f8e-7c76-479b-8511-cb2ed005599a","Type":"ContainerDied","Data":"9b761bfb58a98bfc8c80b929bdb7965b6bc39f1e10932251107f30b8fab8e10c"} Oct 30 15:35:03.290283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.289979 6554 scope.go:122] "RemoveContainer" containerID="3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313" Oct 30 15:35:03.291280 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.291261 6554 generic.go:356] "Generic (PLEG): container finished" podID="f3c19d41-8971-42d6-9406-2fef7d2cec43" containerID="6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66" exitCode=0 Oct 30 15:35:03.291342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.291292 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" event={"ID":"f3c19d41-8971-42d6-9406-2fef7d2cec43","Type":"ContainerDied","Data":"6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66"} Oct 30 15:35:03.291342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.291324 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" Oct 30 15:35:03.291474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.291340 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9" event={"ID":"f3c19d41-8971-42d6-9406-2fef7d2cec43","Type":"ContainerDied","Data":"cd5f5ff8b610347c0a5ac521fb912d768afc95a05348e945ad0641a8a93865c4"} Oct 30 15:35:03.302837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.302822 6554 scope.go:122] "RemoveContainer" containerID="3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313" Oct 30 15:35:03.303103 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:03.303083 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313\": container with ID starting with 3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313 not found: ID does not exist" containerID="3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313" Oct 30 15:35:03.303151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.303110 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313"} err="failed to get container status \"3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313\": rpc error: code = NotFound desc = could not find container \"3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313\": container with ID starting with 3ee92de68c4f190a9a339c93f06c9a4158343fa7c8a3aff4aa2530bcef999313 not found: ID does not exist" Oct 30 15:35:03.303151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.303127 6554 scope.go:122] "RemoveContainer" containerID="6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66" Oct 30 15:35:03.313566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.313538 6554 scope.go:122] "RemoveContainer" containerID="6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66" Oct 30 15:35:03.313803 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:03.313785 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66\": container with ID starting with 6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66 not found: ID does not exist" containerID="6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66" Oct 30 15:35:03.313860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.313807 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66"} err="failed to get container status \"6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66\": rpc error: code = NotFound desc = could not find container \"6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66\": container with ID starting with 6349009da681b0af0742da8b04189e864630885576a4a357d03276ef9f43ca66 not found: ID does not exist" Oct 30 15:35:03.353718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.353693 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh"] Oct 30 15:35:03.367046 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.367019 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5e8bf0-747949df9f-jhfrh"] Oct 30 15:35:03.399115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.399089 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9"] Oct 30 15:35:03.403190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.403167 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5e8bf0-76c8fcd644-grkc9"] Oct 30 15:35:03.525163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.525096 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6"] Oct 30 15:35:03.526090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526064 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f3c19d41-8971-42d6-9406-2fef7d2cec43" containerName="payload-processing" Oct 30 15:35:03.526090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526085 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c19d41-8971-42d6-9406-2fef7d2cec43" containerName="payload-processing" Oct 30 15:35:03.526250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526103 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7b242f8e-7c76-479b-8511-cb2ed005599a" containerName="payload-pre-processing" Oct 30 15:35:03.526250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526108 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b242f8e-7c76-479b-8511-cb2ed005599a" containerName="payload-pre-processing" Oct 30 15:35:03.526250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526157 6554 cpu_manager.go:414] "containerMap: removing container" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" Oct 30 15:35:03.526250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526162 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" Oct 30 15:35:03.526415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526268 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="630789f8-ab60-4ae5-b3b6-5bb64fb62223" containerName="istio-proxy" Oct 30 15:35:03.526415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526276 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c19d41-8971-42d6-9406-2fef7d2cec43" containerName="payload-processing" Oct 30 15:35:03.526415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.526285 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b242f8e-7c76-479b-8511-cb2ed005599a" containerName="payload-pre-processing" Oct 30 15:35:03.531483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.531465 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:03.548613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.548593 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6"] Oct 30 15:35:03.550757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.550736 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-908638\"" Oct 30 15:35:03.600930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.600901 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92"] Oct 30 15:35:03.605552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.605527 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:03.624127 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.624089 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-908638-dockercfg-64xcr\"" Oct 30 15:35:03.624961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.624939 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92"] Oct 30 15:35:03.670408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.654628 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908638-559d97f694-p4md6\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:03.670408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.654862 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4kp\" (UniqueName: \"kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp\") pod \"payload-processing-e2e-ipp-b-908638-559d97f694-p4md6\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:03.788422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.782645 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4kp\" (UniqueName: \"kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp\") pod \"payload-processing-e2e-ipp-b-908638-559d97f694-p4md6\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:03.788422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.782872 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908638-559d97f694-p4md6\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:03.834799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.831172 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-908638-559d97f694-p4md6\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:03.916772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.916054 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwphc\" (UniqueName: \"kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc\") pod \"payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:03.916772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.916318 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:03.920879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:03.920020 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4kp\" (UniqueName: \"kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp\") pod \"payload-processing-e2e-ipp-b-908638-559d97f694-p4md6\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:04.018032 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.017998 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:04.018252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.018235 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwphc\" (UniqueName: \"kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc\") pod \"payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:04.019392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.019340 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:04.031040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.031012 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:35:04.033715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.033611 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:35:04.043384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.043314 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwphc\" (UniqueName: \"kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc\") pod \"payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:04.153549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.153503 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:04.183782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.183751 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b242f8e-7c76-479b-8511-cb2ed005599a" path="/var/lib/kubelet/pods/7b242f8e-7c76-479b-8511-cb2ed005599a/volumes" Oct 30 15:35:04.184553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.184532 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c19d41-8971-42d6-9406-2fef7d2cec43" path="/var/lib/kubelet/pods/f3c19d41-8971-42d6-9406-2fef7d2cec43/volumes" Oct 30 15:35:04.219761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.219550 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:04.475913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.475876 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6"] Oct 30 15:35:04.476421 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:35:04.476394 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f96d2a6_501e_4b37_bc62_1f533b3ea517.slice/crio-0d8be69ac0768fbccb9ba7581f0a484eed40decb70036fe224cfe3dbdde61d0b WatchSource:0}: Error finding container 0d8be69ac0768fbccb9ba7581f0a484eed40decb70036fe224cfe3dbdde61d0b: Status 404 returned error can't find the container with id 0d8be69ac0768fbccb9ba7581f0a484eed40decb70036fe224cfe3dbdde61d0b Oct 30 15:35:04.643627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.643596 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92"] Oct 30 15:35:04.990506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:04.990442 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:35:05.053819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.053790 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:35:05.080730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.080707 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv\" (UniqueName: \"kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv\") pod \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " Oct 30 15:35:05.080877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.080771 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume\") pod \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\" (UID: \"dd409ed2-dcde-47cf-9deb-59ba2121ebed\") " Oct 30 15:35:05.081171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.081149 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume" pod "dd409ed2-dcde-47cf-9deb-59ba2121ebed" (UID: "dd409ed2-dcde-47cf-9deb-59ba2121ebed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:05.081345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.081326 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd409ed2-dcde-47cf-9deb-59ba2121ebed-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:05.083022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.082998 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv" pod "dd409ed2-dcde-47cf-9deb-59ba2121ebed" (UID: "dd409ed2-dcde-47cf-9deb-59ba2121ebed"). InnerVolumeSpecName "kube-api-access-xgvwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:05.182846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.182763 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume\") pod \"27a541bd-a2fd-4488-8695-3de8442c4438\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " Oct 30 15:35:05.182846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.182837 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6\" (UniqueName: \"kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6\") pod \"27a541bd-a2fd-4488-8695-3de8442c4438\" (UID: \"27a541bd-a2fd-4488-8695-3de8442c4438\") " Oct 30 15:35:05.183241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.183212 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume" pod "27a541bd-a2fd-4488-8695-3de8442c4438" (UID: "27a541bd-a2fd-4488-8695-3de8442c4438"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:05.183749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.183720 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27a541bd-a2fd-4488-8695-3de8442c4438-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:05.183892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.183752 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvwv\" (UniqueName: \"kubernetes.io/projected/dd409ed2-dcde-47cf-9deb-59ba2121ebed-kube-api-access-xgvwv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:05.185221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.185192 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6" pod "27a541bd-a2fd-4488-8695-3de8442c4438" (UID: "27a541bd-a2fd-4488-8695-3de8442c4438"). InnerVolumeSpecName "kube-api-access-5d9p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:05.285927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.285896 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d9p6\" (UniqueName: \"kubernetes.io/projected/27a541bd-a2fd-4488-8695-3de8442c4438-kube-api-access-5d9p6\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:05.323323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.322307 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" event={"ID":"2d1db2b4-f1e7-4464-90bd-5829d090abeb","Type":"ContainerStarted","Data":"afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea"} Oct 30 15:35:05.323323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.322389 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" event={"ID":"2d1db2b4-f1e7-4464-90bd-5829d090abeb","Type":"ContainerStarted","Data":"a3b73a6c0e261b53064cac6f0f0d20912d8270e87cd87bd681d4cab0d155a302"} Oct 30 15:35:05.323323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.322422 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:05.324692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.324666 6554 generic.go:356] "Generic (PLEG): container finished" podID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerID="2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc" exitCode=0 Oct 30 15:35:05.324823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.324718 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" Oct 30 15:35:05.324823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.324760 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" event={"ID":"dd409ed2-dcde-47cf-9deb-59ba2121ebed","Type":"ContainerDied","Data":"2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc"} Oct 30 15:35:05.325158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.325132 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz" event={"ID":"dd409ed2-dcde-47cf-9deb-59ba2121ebed","Type":"ContainerDied","Data":"9af91660c5a0e406c633ba696cb190b14baa8d6ff61bdc70144ea25baaa96919"} Oct 30 15:35:05.325270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.325205 6554 scope.go:122] "RemoveContainer" containerID="2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc" Oct 30 15:35:05.326927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.326902 6554 generic.go:356] "Generic (PLEG): container finished" podID="27a541bd-a2fd-4488-8695-3de8442c4438" containerID="3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26" exitCode=0 Oct 30 15:35:05.327080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.327060 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" event={"ID":"27a541bd-a2fd-4488-8695-3de8442c4438","Type":"ContainerDied","Data":"3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26"} Oct 30 15:35:05.327170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.327104 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" event={"ID":"27a541bd-a2fd-4488-8695-3de8442c4438","Type":"ContainerDied","Data":"0fd0453c6ff37cd5f7bbb38a47c364b6f00761c944ecdb407f2a866641ed0e3a"} Oct 30 15:35:05.327231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.327026 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw" Oct 30 15:35:05.338705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.338026 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" event={"ID":"2f96d2a6-501e-4b37-bc62-1f533b3ea517","Type":"ContainerStarted","Data":"d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241"} Oct 30 15:35:05.338705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.338074 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" event={"ID":"2f96d2a6-501e-4b37-bc62-1f533b3ea517","Type":"ContainerStarted","Data":"0d8be69ac0768fbccb9ba7581f0a484eed40decb70036fe224cfe3dbdde61d0b"} Oct 30 15:35:05.338705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.338277 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:05.340780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.340762 6554 scope.go:122] "RemoveContainer" containerID="2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc" Oct 30 15:35:05.341083 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:05.341064 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc\": container with ID starting with 2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc not found: ID does not exist" containerID="2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc" Oct 30 15:35:05.341145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.341090 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc"} err="failed to get container status \"2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc\": rpc error: code = NotFound desc = could not find container \"2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc\": container with ID starting with 2b1bbb268aa745e2dcf3adcd8dce6ff6da007fe4063f401211e73bd9ec692bfc not found: ID does not exist" Oct 30 15:35:05.341145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.341105 6554 scope.go:122] "RemoveContainer" containerID="3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26" Oct 30 15:35:05.351012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.350993 6554 scope.go:122] "RemoveContainer" containerID="3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26" Oct 30 15:35:05.351270 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:05.351251 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26\": container with ID starting with 3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26 not found: ID does not exist" containerID="3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26" Oct 30 15:35:05.351337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.351275 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26"} err="failed to get container status \"3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26\": rpc error: code = NotFound desc = could not find container \"3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26\": container with ID starting with 3627fb864111fbd030985b8c7c90d3d19b131e81b5fa62d22b0a90000a9d4b26 not found: ID does not exist" Oct 30 15:35:05.376062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.376026 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" podStartSLOduration=2.376011727 podStartE2EDuration="2.376011727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:05.374838624 +0000 UTC m=+2809.333880043" watchObservedRunningTime="2026-10-30 15:35:05.376011727 +0000 UTC m=+2809.335053133" Oct 30 15:35:05.395058 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:05.395031 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd409ed2_dcde_47cf_9deb_59ba2121ebed.slice/crio-9af91660c5a0e406c633ba696cb190b14baa8d6ff61bdc70144ea25baaa96919\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27a541bd_a2fd_4488_8695_3de8442c4438.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd409ed2_dcde_47cf_9deb_59ba2121ebed.slice\": RecentStats: unable to find data in memory cache]" Oct 30 15:35:05.416730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.416707 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw"] Oct 30 15:35:05.422623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.422603 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-416d92-6bd67bc6bc-fmmkw"] Oct 30 15:35:05.461487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.461417 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" podStartSLOduration=2.46140648 podStartE2EDuration="2.46140648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:05.460800838 +0000 UTC m=+2809.419842244" watchObservedRunningTime="2026-10-30 15:35:05.46140648 +0000 UTC m=+2809.420447876" Oct 30 15:35:05.502512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.502493 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz"] Oct 30 15:35:05.505644 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:05.505624 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-416d92-64d659bd78-j89bz"] Oct 30 15:35:06.156117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:06.156084 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" path="/var/lib/kubelet/pods/27a541bd-a2fd-4488-8695-3de8442c4438/volumes" Oct 30 15:35:06.156535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:06.156520 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" path="/var/lib/kubelet/pods/dd409ed2-dcde-47cf-9deb-59ba2121ebed/volumes" Oct 30 15:35:09.204023 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:09.203980 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:35:16.352417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:16.352386 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:35:16.353023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:16.352997 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:35:16.391973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:16.391938 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h"] Oct 30 15:35:16.392213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:16.392185 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" containerID="cri-o://7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0" gracePeriod=30 Oct 30 15:35:20.472210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:20.472178 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:35:20.472636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:20.472236 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:35:21.685484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.685459 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:35:21.754661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754633 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.754834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754713 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.754834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754742 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.754834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754775 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.755016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754908 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.755016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754953 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.755016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.754987 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.755180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.755084 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.755180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.755129 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy\") pod \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\" (UID: \"2d982a69-0c73-4b3d-9e1b-388309b69ffd\") " Oct 30 15:35:21.755941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.755915 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:35:21.756374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.756337 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.756537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.756452 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.756714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.756690 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.756845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.756805 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.763520 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.762860 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:21.764429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.763742 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:35:21.768395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.765100 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:35:21.768395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.765169 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn" pod "2d982a69-0c73-4b3d-9e1b-388309b69ffd" (UID: "2d982a69-0c73-4b3d-9e1b-388309b69ffd"). InnerVolumeSpecName "kube-api-access-2w7tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863822 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863864 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863877 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863889 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w7tn\" (UniqueName: \"kubernetes.io/projected/2d982a69-0c73-4b3d-9e1b-388309b69ffd-kube-api-access-2w7tn\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863904 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863917 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863930 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863942 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:21.866370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:21.863954 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d982a69-0c73-4b3d-9e1b-388309b69ffd-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:35:22.488733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.488657 6554 generic.go:356] "Generic (PLEG): container finished" podID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerID="7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0" exitCode=0 Oct 30 15:35:22.488733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.488709 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" Oct 30 15:35:22.488940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.488728 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" event={"ID":"2d982a69-0c73-4b3d-9e1b-388309b69ffd","Type":"ContainerDied","Data":"7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0"} Oct 30 15:35:22.488940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.488779 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h" event={"ID":"2d982a69-0c73-4b3d-9e1b-388309b69ffd","Type":"ContainerDied","Data":"7accbccb7010982118ce0343a5230f428f49af6aa69ada6bd09b78f57c6d7d52"} Oct 30 15:35:22.488940 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.488801 6554 scope.go:122] "RemoveContainer" containerID="7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0" Oct 30 15:35:22.499955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.499925 6554 scope.go:122] "RemoveContainer" containerID="7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0" Oct 30 15:35:22.500274 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:22.500219 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0\": container with ID starting with 7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0 not found: ID does not exist" containerID="7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0" Oct 30 15:35:22.500274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.500252 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0"} err="failed to get container status \"7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0\": rpc error: code = NotFound desc = could not find container \"7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0\": container with ID starting with 7665fe08a191da61659d19af7d6f882847d24150d95e53ab140cb840b3e573f0 not found: ID does not exist" Oct 30 15:35:22.560610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.560585 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h"] Oct 30 15:35:22.564913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:22.564890 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-416d92-openshift-default-68b875f7f7-jq27h"] Oct 30 15:35:24.156579 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:24.156547 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" path="/var/lib/kubelet/pods/2d982a69-0c73-4b3d-9e1b-388309b69ffd/volumes" Oct 30 15:35:39.202904 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:35:39.202867 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:35:49.298278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.298235 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2"] Oct 30 15:35:49.299261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299236 6554 cpu_manager.go:414] "containerMap: removing container" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" containerName="payload-pre-processing" Oct 30 15:35:49.299403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299261 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" containerName="payload-pre-processing" Oct 30 15:35:49.299403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299289 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" Oct 30 15:35:49.299403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299297 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" Oct 30 15:35:49.299403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299328 6554 cpu_manager.go:414] "containerMap: removing container" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerName="payload-processing" Oct 30 15:35:49.299403 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299336 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerName="payload-processing" Oct 30 15:35:49.299688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299585 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="27a541bd-a2fd-4488-8695-3de8442c4438" containerName="payload-pre-processing" Oct 30 15:35:49.299688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299607 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d982a69-0c73-4b3d-9e1b-388309b69ffd" containerName="istio-proxy" Oct 30 15:35:49.299688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.299622 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd409ed2-dcde-47cf-9deb-59ba2121ebed" containerName="payload-processing" Oct 30 15:35:49.303564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.303543 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.330878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.330822 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-0644a3\"" Oct 30 15:35:49.330878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.330822 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-0644a3-dockercfg-nrchf\"" Oct 30 15:35:49.340221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.340193 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2"] Oct 30 15:35:49.391290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.391261 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt"] Oct 30 15:35:49.395944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.395922 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.431209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.431173 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt"] Oct 30 15:35:49.477760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.471882 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48hfg\" (UniqueName: \"kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.478261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.478226 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.615428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.613120 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48hfg\" (UniqueName: \"kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.615428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.613310 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlj4b\" (UniqueName: \"kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b\") pod \"payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.615428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.613390 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.615428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.613478 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.615428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.614780 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.642030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.641999 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48hfg\" (UniqueName: \"kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg\") pod \"payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:49.714941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.714907 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlj4b\" (UniqueName: \"kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b\") pod \"payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.715260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.715010 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.715860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.715834 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.739583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.739556 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlj4b\" (UniqueName: \"kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b\") pod \"payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:49.919067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:49.919044 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:50.034462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.034428 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:50.233904 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:35:50.233872 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb667d5d7_3ff2_4114_b315_184883aa521e.slice/crio-7aafc4f4ff7cd11b5dba710a82b724e38cb58fc2d1dee8cb7956b8652dfe6ae9 WatchSource:0}: Error finding container 7aafc4f4ff7cd11b5dba710a82b724e38cb58fc2d1dee8cb7956b8652dfe6ae9: Status 404 returned error can't find the container with id 7aafc4f4ff7cd11b5dba710a82b724e38cb58fc2d1dee8cb7956b8652dfe6ae9 Oct 30 15:35:50.234270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.234247 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2"] Oct 30 15:35:50.351251 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:35:50.351226 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6492473e_58d4_4b90_bd3d_b1fcd157e6cb.slice/crio-858e6c7c5783a1e853e99e8a838083c39d27c4ef17ad42b08702cf7a5ccf91c7 WatchSource:0}: Error finding container 858e6c7c5783a1e853e99e8a838083c39d27c4ef17ad42b08702cf7a5ccf91c7: Status 404 returned error can't find the container with id 858e6c7c5783a1e853e99e8a838083c39d27c4ef17ad42b08702cf7a5ccf91c7 Oct 30 15:35:50.352162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.352140 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt"] Oct 30 15:35:50.759690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.759657 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" event={"ID":"b667d5d7-3ff2-4114-b315-184883aa521e","Type":"ContainerStarted","Data":"a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57"} Oct 30 15:35:50.759858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.759707 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" event={"ID":"b667d5d7-3ff2-4114-b315-184883aa521e","Type":"ContainerStarted","Data":"7aafc4f4ff7cd11b5dba710a82b724e38cb58fc2d1dee8cb7956b8652dfe6ae9"} Oct 30 15:35:50.759929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.759858 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:35:50.761417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.761385 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" event={"ID":"6492473e-58d4-4b90-bd3d-b1fcd157e6cb","Type":"ContainerStarted","Data":"71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e"} Oct 30 15:35:50.761570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.761433 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" event={"ID":"6492473e-58d4-4b90-bd3d-b1fcd157e6cb","Type":"ContainerStarted","Data":"858e6c7c5783a1e853e99e8a838083c39d27c4ef17ad42b08702cf7a5ccf91c7"} Oct 30 15:35:50.761570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.761486 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:35:50.807797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.807749 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" podStartSLOduration=1.807737531 podStartE2EDuration="1.807737531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:50.805010106 +0000 UTC m=+2854.764051525" watchObservedRunningTime="2026-10-30 15:35:50.807737531 +0000 UTC m=+2854.766778935" Oct 30 15:35:50.851906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:50.851865 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" podStartSLOduration=1.851854323 podStartE2EDuration="1.851854323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:35:50.849320349 +0000 UTC m=+2854.808361753" watchObservedRunningTime="2026-10-30 15:35:50.851854323 +0000 UTC m=+2854.810895734" Oct 30 15:35:57.296336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:57.296301 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm"] Oct 30 15:35:57.296821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:35:57.296523 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" containerID="cri-o://ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323" gracePeriod=30 Oct 30 15:36:01.771313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:01.771281 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:36:01.771927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:01.771397 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:36:02.572825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.572767 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:36:02.752844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.751887 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.752844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.752712 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.753259 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.753227 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.754136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.753808 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:02.754281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754233 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:02.754281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754264 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:02.754472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754400 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.754603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754584 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.754748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754628 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.754748 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754688 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.754867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754755 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.754932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754898 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:02.754932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.754906 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy\") pod \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\" (UID: \"ec9e976d-ce6e-48a4-a171-a46075af6ed1\") " Oct 30 15:36:02.755162 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755124 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:02.755727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755702 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.755825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755733 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.755825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755738 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:02.755825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755748 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.755825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755792 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.755825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.755807 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.756861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.756837 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:36:02.756952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.756907 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:02.757164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.757149 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs" pod "ec9e976d-ce6e-48a4-a171-a46075af6ed1" (UID: "ec9e976d-ce6e-48a4-a171-a46075af6ed1"). InnerVolumeSpecName "kube-api-access-bw2qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:02.854494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.854436 6554 generic.go:356] "Generic (PLEG): container finished" podID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerID="ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323" exitCode=0 Oct 30 15:36:02.854494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.854484 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" Oct 30 15:36:02.854921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.854523 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" event={"ID":"ec9e976d-ce6e-48a4-a171-a46075af6ed1","Type":"ContainerDied","Data":"ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323"} Oct 30 15:36:02.854921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.854591 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm" event={"ID":"ec9e976d-ce6e-48a4-a171-a46075af6ed1","Type":"ContainerDied","Data":"2fed1783dac8a35344fbc7330b3426c5ea9cb9372458b87d65fd02e40185abf9"} Oct 30 15:36:02.854921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.854626 6554 scope.go:122] "RemoveContainer" containerID="ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323" Oct 30 15:36:02.856859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.856839 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.856859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.856860 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw2qs\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-kube-api-access-bw2qs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.856859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.856871 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.857098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.856880 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec9e976d-ce6e-48a4-a171-a46075af6ed1-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:02.866091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.866075 6554 scope.go:122] "RemoveContainer" containerID="ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323" Oct 30 15:36:02.866387 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:02.866359 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323\": container with ID starting with ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323 not found: ID does not exist" containerID="ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323" Oct 30 15:36:02.866450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.866394 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323"} err="failed to get container status \"ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323\": rpc error: code = NotFound desc = could not find container \"ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323\": container with ID starting with ea730a08c6f41da77a2333df5184a30ae3214bd6acdc928c6c7ca423ed28e323 not found: ID does not exist" Oct 30 15:36:02.951523 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.951500 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm"] Oct 30 15:36:02.966045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:02.966022 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-2jsbm"] Oct 30 15:36:04.155920 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:04.155889 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" path="/var/lib/kubelet/pods/ec9e976d-ce6e-48a4-a171-a46075af6ed1/volumes" Oct 30 15:36:06.067189 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.067155 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx"] Oct 30 15:36:06.068199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.068174 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" Oct 30 15:36:06.068199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.068199 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" Oct 30 15:36:06.068453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.068418 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9e976d-ce6e-48a4-a171-a46075af6ed1" containerName="istio-proxy" Oct 30 15:36:06.073904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.073871 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.098612 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.098582 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx"] Oct 30 15:36:06.106726 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.106706 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e30f62-openshift-default-dockercfg-npg4l\"" Oct 30 15:36:06.200923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.200888 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.200923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.200922 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.200949 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.201061 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.201113 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjz9c\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.201146 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.201215 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.201343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.201496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.201480 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309068 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309301 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309516 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309619 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309649 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309688 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309759 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309790 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjz9c\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.309825 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.310402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.310283 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.335417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.332933 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.335417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.334148 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.339374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.336253 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.354374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.352909 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.381036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.381001 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.381268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.381242 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.446370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.436564 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjz9c\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.446370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.438807 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token\") pod \"e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:06.691510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:06.691467 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:07.019847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.019816 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx"] Oct 30 15:36:07.019984 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:36:07.019962 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode58e5fda_c858_4072_8631_307319269c5e.slice/crio-101982bb2db41151cad4dad928be65413fb6bb37b0a34ec71da38d4d9f5637f4 WatchSource:0}: Error finding container 101982bb2db41151cad4dad928be65413fb6bb37b0a34ec71da38d4d9f5637f4: Status 404 returned error can't find the container with id 101982bb2db41151cad4dad928be65413fb6bb37b0a34ec71da38d4d9f5637f4 Oct 30 15:36:07.023036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.023000 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:36:07.023130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.023108 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:36:07.023185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.023149 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:36:07.894547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.894479 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" event={"ID":"e58e5fda-c858-4072-8631-307319269c5e","Type":"ContainerStarted","Data":"11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c"} Oct 30 15:36:07.894547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.894539 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" event={"ID":"e58e5fda-c858-4072-8631-307319269c5e","Type":"ContainerStarted","Data":"101982bb2db41151cad4dad928be65413fb6bb37b0a34ec71da38d4d9f5637f4"} Oct 30 15:36:07.968805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:07.968756 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podStartSLOduration=1.968740597 podStartE2EDuration="1.968740597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:07.965449314 +0000 UTC m=+2871.924490732" watchObservedRunningTime="2026-10-30 15:36:07.968740597 +0000 UTC m=+2871.927782004" Oct 30 15:36:08.692257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:08.692219 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:08.698680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:08.698655 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:08.901283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:08.901254 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:08.902108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:08.902088 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:36:09.378718 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:09.378675 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:36:12.737517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:12.737477 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7"] Oct 30 15:36:13.092062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.082445 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.100442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.099506 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7"] Oct 30 15:36:13.113497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.110200 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f3a578-openshift-default-dockercfg-hwx69\"" Oct 30 15:36:13.128209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.118119 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5"] Oct 30 15:36:13.128209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.128022 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5"] Oct 30 15:36:13.128209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.128176 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.137039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.133611 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj"] Oct 30 15:36:13.146657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.143166 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj"] Oct 30 15:36:13.146657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.143313 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.146657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.145809 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-e30f62\"" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.177770 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.177868 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spbvq\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.177934 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.177955 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.177979 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178030 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g22xd\" (UniqueName: \"kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178064 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178158 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178219 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178255 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstkj\" (UniqueName: \"kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj\") pod \"payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178271 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178293 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.179130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.178313 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.208705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.208675 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-e30f62-dockercfg-k4fqw\"" Oct 30 15:36:13.280680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280649 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spbvq\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.280867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280698 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.280867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280720 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.280867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280744 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.280867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280785 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g22xd\" (UniqueName: \"kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.280867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280848 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280889 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280946 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.280985 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstkj\" (UniqueName: \"kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj\") pod \"payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.281178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281126 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281163 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.281537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281198 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281274 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281314 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.281537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281375 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281197 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281511 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281863 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.281973 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.281954 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.282252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.282231 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.283470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.283444 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.283735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.283709 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.305769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.305743 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.305905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.305775 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g22xd\" (UniqueName: \"kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd\") pod \"payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.306091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.306071 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstkj\" (UniqueName: \"kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj\") pod \"payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.306158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.306081 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spbvq\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq\") pod \"e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.439494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.439475 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:13.457495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.457472 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:13.479975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.479952 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:13.775123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.775099 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7"] Oct 30 15:36:13.775540 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:36:13.775439 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83390336_5acc_4b7e_961f_ddb81298a3ba.slice/crio-1f6ac489c4e6d8ed7bd0e152d872ec6ede0104c3534a33a46ceb213c07bf147c WatchSource:0}: Error finding container 1f6ac489c4e6d8ed7bd0e152d872ec6ede0104c3534a33a46ceb213c07bf147c: Status 404 returned error can't find the container with id 1f6ac489c4e6d8ed7bd0e152d872ec6ede0104c3534a33a46ceb213c07bf147c Oct 30 15:36:13.778438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.778403 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:36:13.778536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.778473 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:36:13.778536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.778503 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:36:13.794257 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:36:13.794230 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb2d6c6_35eb_45de_a356_23f3369c68a5.slice/crio-4e71a75a5df007db27f9267efb73267d0281dc04b6fc6eccb4074aefee8eeafa WatchSource:0}: Error finding container 4e71a75a5df007db27f9267efb73267d0281dc04b6fc6eccb4074aefee8eeafa: Status 404 returned error can't find the container with id 4e71a75a5df007db27f9267efb73267d0281dc04b6fc6eccb4074aefee8eeafa Oct 30 15:36:13.794926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.794900 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5"] Oct 30 15:36:13.921680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:13.921651 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj"] Oct 30 15:36:13.923781 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:36:13.923754 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccd772a_10fc_48f8_bcc5_27e24af681f4.slice/crio-9fee82821cbb5214e637c9a4c0f59bbaae90fa522f320a64fd4b9782cd62bd78 WatchSource:0}: Error finding container 9fee82821cbb5214e637c9a4c0f59bbaae90fa522f320a64fd4b9782cd62bd78: Status 404 returned error can't find the container with id 9fee82821cbb5214e637c9a4c0f59bbaae90fa522f320a64fd4b9782cd62bd78 Oct 30 15:36:14.101699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.101607 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" event={"ID":"6cb2d6c6-35eb-45de-a356-23f3369c68a5","Type":"ContainerStarted","Data":"400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85"} Oct 30 15:36:14.101699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.101667 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" event={"ID":"6cb2d6c6-35eb-45de-a356-23f3369c68a5","Type":"ContainerStarted","Data":"4e71a75a5df007db27f9267efb73267d0281dc04b6fc6eccb4074aefee8eeafa"} Oct 30 15:36:14.101699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.101694 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:14.103889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.103864 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" event={"ID":"83390336-5acc-4b7e-961f-ddb81298a3ba","Type":"ContainerStarted","Data":"7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9"} Oct 30 15:36:14.104032 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.103909 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" event={"ID":"83390336-5acc-4b7e-961f-ddb81298a3ba","Type":"ContainerStarted","Data":"1f6ac489c4e6d8ed7bd0e152d872ec6ede0104c3534a33a46ceb213c07bf147c"} Oct 30 15:36:14.105566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.105533 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" event={"ID":"4ccd772a-10fc-48f8-bcc5-27e24af681f4","Type":"ContainerStarted","Data":"bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1"} Oct 30 15:36:14.105670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.105593 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:14.105670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.105616 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" event={"ID":"4ccd772a-10fc-48f8-bcc5-27e24af681f4","Type":"ContainerStarted","Data":"9fee82821cbb5214e637c9a4c0f59bbaae90fa522f320a64fd4b9782cd62bd78"} Oct 30 15:36:14.148120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.148077 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" podStartSLOduration=2.148065962 podStartE2EDuration="2.148065962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:14.147663862 +0000 UTC m=+2878.106705271" watchObservedRunningTime="2026-10-30 15:36:14.148065962 +0000 UTC m=+2878.107107367" Oct 30 15:36:14.194100 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.194046 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" podStartSLOduration=2.194028079 podStartE2EDuration="2.194028079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:14.191035736 +0000 UTC m=+2878.150077146" watchObservedRunningTime="2026-10-30 15:36:14.194028079 +0000 UTC m=+2878.153069485" Oct 30 15:36:14.240874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.240825 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podStartSLOduration=2.240812087 podStartE2EDuration="2.240812087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:36:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:36:14.2385991 +0000 UTC m=+2878.197640532" watchObservedRunningTime="2026-10-30 15:36:14.240812087 +0000 UTC m=+2878.199853492" Oct 30 15:36:14.440967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.440929 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:14.442720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.442689 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:36:14.442859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:14.442738 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 15:36:15.439948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:15.439913 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:36:15.440374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:15.439977 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 15:36:16.441135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:16.440882 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:36:16.441135 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:16.440945 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 15:36:17.445499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:17.445455 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:36:17.446070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:17.445518 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 15:36:18.440875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:18.440846 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:36:18.441046 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:18.440894 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 15:36:19.440575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:19.440543 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body= Oct 30 15:36:19.440994 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:19.440595 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" Oct 30 15:36:20.566705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:20.566674 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:20.567331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:20.567057 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:20.568751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:20.568730 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:36:23.357437 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:23.357399 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd"] Oct 30 15:36:23.357857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:23.357596 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" containerID="cri-o://746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5" gracePeriod=30 Oct 30 15:36:25.116575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:25.116548 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:36:25.117059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:25.116884 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:36:28.646481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.646456 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:36:28.739474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739445 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.739637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739561 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.739711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739659 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.739783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739729 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.739783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739742 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:28.739898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739787 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.739898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739891 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.740011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739919 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.740011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.739962 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.740635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.740608 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:28.740815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.740785 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:28.740951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.740929 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:28.741059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741007 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:28.741059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741043 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs\") pod \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\" (UID: \"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889\") " Oct 30 15:36:28.741843 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741822 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.741959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741847 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.741959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741861 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.741959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741873 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.741959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.741886 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.742495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.742461 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:28.745436 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.745406 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "kube-api-access-9n5nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:28.745545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.745469 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:36:28.746754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.746723 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy" pod "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" (UID: "ac2ec6b9-f6e2-418c-baee-f6dcd25e2889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:28.842957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.842930 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.842957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.842955 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.843118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.842964 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:28.843118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:28.842974 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n5nm\" (UniqueName: \"kubernetes.io/projected/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889-kube-api-access-9n5nm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:29.222961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.222920 6554 generic.go:356] "Generic (PLEG): container finished" podID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerID="746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5" exitCode=0 Oct 30 15:36:29.223160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.222992 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" Oct 30 15:36:29.223160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.223001 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" event={"ID":"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889","Type":"ContainerDied","Data":"746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5"} Oct 30 15:36:29.223160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.223055 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd" event={"ID":"ac2ec6b9-f6e2-418c-baee-f6dcd25e2889","Type":"ContainerDied","Data":"d61fcb445f0924ad6dcb097949a1d6aafd6764fd484ddac48285108bda83288e"} Oct 30 15:36:29.223160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.223079 6554 scope.go:122] "RemoveContainer" containerID="746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5" Oct 30 15:36:29.235261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.235242 6554 scope.go:122] "RemoveContainer" containerID="7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df" Oct 30 15:36:29.245524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.245492 6554 scope.go:122] "RemoveContainer" containerID="746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5" Oct 30 15:36:29.245790 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:29.245766 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5\": container with ID starting with 746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5 not found: ID does not exist" containerID="746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5" Oct 30 15:36:29.245870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.245800 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5"} err="failed to get container status \"746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5\": rpc error: code = NotFound desc = could not find container \"746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5\": container with ID starting with 746a150271c25f0b5809d76df2ba838bcf6e61c8f08bdda3c4e3444e79295bf5 not found: ID does not exist" Oct 30 15:36:29.245870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.245821 6554 scope.go:122] "RemoveContainer" containerID="7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df" Oct 30 15:36:29.246091 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:29.246072 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df\": container with ID starting with 7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df not found: ID does not exist" containerID="7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df" Oct 30 15:36:29.246132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.246101 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df"} err="failed to get container status \"7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df\": rpc error: code = NotFound desc = could not find container \"7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df\": container with ID starting with 7e6c48f1b3d6c5f51b8a69e691dfdf5f087dd03b525257a87d12a16a2513f2df not found: ID does not exist" Oct 30 15:36:29.296147 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.296122 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd"] Oct 30 15:36:29.309324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:29.309296 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf437b-openshift-default-6687d8cd47-x6jtd"] Oct 30 15:36:30.160731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:30.160697 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" path="/var/lib/kubelet/pods/ac2ec6b9-f6e2-418c-baee-f6dcd25e2889/volumes" Oct 30 15:36:34.162454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:34.162416 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb"] Oct 30 15:36:34.162967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:34.162676 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" containerID="cri-o://4d7b3fdd1b0f4b2adafe505c4a5303cd024bac869e1dd1292646c741f1ffd86c" gracePeriod=30 Oct 30 15:36:39.204007 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:39.203970 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:36:39.357613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.357503 6554 generic.go:356] "Generic (PLEG): container finished" podID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerID="4d7b3fdd1b0f4b2adafe505c4a5303cd024bac869e1dd1292646c741f1ffd86c" exitCode=0 Oct 30 15:36:39.357613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.357551 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" event={"ID":"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5","Type":"ContainerDied","Data":"4d7b3fdd1b0f4b2adafe505c4a5303cd024bac869e1dd1292646c741f1ffd86c"} Oct 30 15:36:39.447283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.447220 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:36:39.622735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622705 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.622922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622742 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.622922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622778 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.622922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622811 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.622922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622894 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.623161 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622925 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.623161 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.622981 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.623161 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623100 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.623325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623176 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs\") pod \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\" (UID: \"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5\") " Oct 30 15:36:39.623402 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623323 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.623586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623531 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.623586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623577 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.623743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623685 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:36:39.623867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.623832 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.624165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.624138 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.624276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.624171 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.624276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.624188 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.624276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.624202 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.624276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.624226 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.625214 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.625187 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:39.625338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.625264 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "kube-api-access-f682c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:39.625499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.625479 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:36:39.625686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.625660 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy" pod "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" (UID: "0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:39.643329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.643304 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25"] Oct 30 15:36:39.655260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.655236 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv"] Oct 30 15:36:39.725211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.725151 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.725211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.725179 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f682c\" (UniqueName: \"kubernetes.io/projected/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-kube-api-access-f682c\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.725211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.725190 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:39.725211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:39.725199 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:40.366172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:40.366132 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" Oct 30 15:36:40.366732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:40.366190 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb" event={"ID":"0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5","Type":"ContainerDied","Data":"02106d6b24df3777b20d9fd9416a374559a2cb95537ac429f03191a0514a938f"} Oct 30 15:36:40.366732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:40.366244 6554 scope.go:122] "RemoveContainer" containerID="4d7b3fdd1b0f4b2adafe505c4a5303cd024bac869e1dd1292646c741f1ffd86c" Oct 30 15:36:40.438875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:40.438822 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb"] Oct 30 15:36:40.452087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:40.452063 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-100b2e-openshift-default-99b89c967-9jzfb"] Oct 30 15:36:42.156925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:42.156894 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" path="/var/lib/kubelet/pods/0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5/volumes" Oct 30 15:36:45.688039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:45.687996 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46"] Oct 30 15:36:50.688626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:50.688591 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" containerName="payload-processing" containerID="cri-o://bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f" gracePeriod=25 Oct 30 15:36:54.202929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:54.202896 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:36:54.665896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:54.665861 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" containerID="cri-o://61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc" gracePeriod=45 Oct 30 15:36:54.679780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:54.679752 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" containerID="cri-o://1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c" gracePeriod=45 Oct 30 15:36:54.712632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:54.712607 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": dial tcp 10.134.0.73:8000: connect: connection refused" Oct 30 15:36:54.989176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:54.989146 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:36:54.993087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:54.993066 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:36:55.041430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041399 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s\" (UniqueName: \"kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s\") pod \"05fd71f9-b19f-4a9d-8c80-e2097316161d\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " Oct 30 15:36:55.041572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041468 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm\") pod \"05fd71f9-b19f-4a9d-8c80-e2097316161d\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " Oct 30 15:36:55.041572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041514 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache\") pod \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041570 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir\") pod \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041627 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb\" (UniqueName: \"kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb\") pod \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041731 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home\") pod \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041761 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir\") pod \"05fd71f9-b19f-4a9d-8c80-e2097316161d\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041801 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir" pod "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" (UID: "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041823 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache" pod "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" (UID: "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041856 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home\") pod \"05fd71f9-b19f-4a9d-8c80-e2097316161d\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041910 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs\") pod \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.041940 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm\") pod \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\" (UID: \"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a\") " Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042018 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home" pod "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" (UID: "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.042133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042032 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache\") pod \"05fd71f9-b19f-4a9d-8c80-e2097316161d\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " Oct 30 15:36:55.042802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042136 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home" pod "05fd71f9-b19f-4a9d-8c80-e2097316161d" (UID: "05fd71f9-b19f-4a9d-8c80-e2097316161d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.042802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042164 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs\" (UniqueName: \"kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs\") pod \"05fd71f9-b19f-4a9d-8c80-e2097316161d\" (UID: \"05fd71f9-b19f-4a9d-8c80-e2097316161d\") " Oct 30 15:36:55.042802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042242 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir" pod "05fd71f9-b19f-4a9d-8c80-e2097316161d" (UID: "05fd71f9-b19f-4a9d-8c80-e2097316161d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.042802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042462 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache" pod "05fd71f9-b19f-4a9d-8c80-e2097316161d" (UID: "05fd71f9-b19f-4a9d-8c80-e2097316161d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.043029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.042980 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.043029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.043003 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.043029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.043015 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.043029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.043026 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.043255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.043037 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.043255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.043049 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.044099 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.044071 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s" pod "05fd71f9-b19f-4a9d-8c80-e2097316161d" (UID: "05fd71f9-b19f-4a9d-8c80-e2097316161d"). InnerVolumeSpecName "kube-api-access-khk9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:55.045446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.045411 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm" pod "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" (UID: "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.045690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.045629 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs" pod "05fd71f9-b19f-4a9d-8c80-e2097316161d" (UID: "05fd71f9-b19f-4a9d-8c80-e2097316161d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:55.046327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.046302 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm" pod "05fd71f9-b19f-4a9d-8c80-e2097316161d" (UID: "05fd71f9-b19f-4a9d-8c80-e2097316161d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:36:55.048454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.048426 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs" pod "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" (UID: "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:36:55.048706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.048671 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb" pod "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" (UID: "f6b0c106-9b7e-4f7b-9aea-15bf5272a65a"). InnerVolumeSpecName "kube-api-access-hmttb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:36:55.151711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.145883 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.151711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.145919 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.151711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.145934 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05fd71f9-b19f-4a9d-8c80-e2097316161d-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.151711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.145948 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khk9s\" (UniqueName: \"kubernetes.io/projected/05fd71f9-b19f-4a9d-8c80-e2097316161d-kube-api-access-khk9s\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.151711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.145972 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05fd71f9-b19f-4a9d-8c80-e2097316161d-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.151711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.145986 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmttb\" (UniqueName: \"kubernetes.io/projected/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a-kube-api-access-hmttb\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:36:55.521184 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.521151 6554 generic.go:356] "Generic (PLEG): container finished" podID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerID="61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc" exitCode=0 Oct 30 15:36:55.521690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.521206 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" Oct 30 15:36:55.521690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.521242 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" event={"ID":"05fd71f9-b19f-4a9d-8c80-e2097316161d","Type":"ContainerDied","Data":"61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc"} Oct 30 15:36:55.521690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.521286 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25" event={"ID":"05fd71f9-b19f-4a9d-8c80-e2097316161d","Type":"ContainerDied","Data":"e4173517216d5d399c9364bc01e168e137729c1a3b0124c98d4f5282457dbd7d"} Oct 30 15:36:55.521690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.521305 6554 scope.go:122] "RemoveContainer" containerID="61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc" Oct 30 15:36:55.523023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.522987 6554 generic.go:356] "Generic (PLEG): container finished" podID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerID="1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c" exitCode=0 Oct 30 15:36:55.523165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.523047 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" Oct 30 15:36:55.523165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.523074 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" event={"ID":"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a","Type":"ContainerDied","Data":"1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c"} Oct 30 15:36:55.523165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.523105 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv" event={"ID":"f6b0c106-9b7e-4f7b-9aea-15bf5272a65a","Type":"ContainerDied","Data":"00076d9ee34a7fbc8496bfe8cefadafc76e42a14f9af614eb4b144e4f07fcf99"} Oct 30 15:36:55.532711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.532517 6554 scope.go:122] "RemoveContainer" containerID="61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc" Oct 30 15:36:55.532874 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:55.532850 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc\": container with ID starting with 61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc not found: ID does not exist" containerID="61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc" Oct 30 15:36:55.532960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.532885 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc"} err="failed to get container status \"61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc\": rpc error: code = NotFound desc = could not find container \"61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc\": container with ID starting with 61614ca3e54190d2ba2801274d1e9743d72c342b21dc4cd9629fa58f8c38c9fc not found: ID does not exist" Oct 30 15:36:55.532960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.532924 6554 scope.go:122] "RemoveContainer" containerID="1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c" Oct 30 15:36:55.542192 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.542176 6554 scope.go:122] "RemoveContainer" containerID="1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c" Oct 30 15:36:55.542476 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:36:55.542455 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c\": container with ID starting with 1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c not found: ID does not exist" containerID="1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c" Oct 30 15:36:55.542558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.542483 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c"} err="failed to get container status \"1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c\": rpc error: code = NotFound desc = could not find container \"1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c\": container with ID starting with 1f6a4942a17d5f3d548bc4105736492e2431ef816d6a3b5f4ddc912c6bad475c not found: ID does not exist" Oct 30 15:36:55.581194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.581176 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv"] Oct 30 15:36:55.587809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.587785 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/premium-w5-4f5c30-kserve-d79d9dddb-h84fv"] Oct 30 15:36:55.624178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.624155 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25"] Oct 30 15:36:55.628343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:55.628322 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-4f5c30/distinct-w5-4f5c30-kserve-55b7b79749-tvb25"] Oct 30 15:36:56.156492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:56.156464 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" path="/var/lib/kubelet/pods/05fd71f9-b19f-4a9d-8c80-e2097316161d/volumes" Oct 30 15:36:56.156931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:36:56.156915 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" path="/var/lib/kubelet/pods/f6b0c106-9b7e-4f7b-9aea-15bf5272a65a/volumes" Oct 30 15:37:03.714188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:03.714156 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6"] Oct 30 15:37:03.748233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:03.748204 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92"] Oct 30 15:37:04.203898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:04.203857 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:37:08.715164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:08.715114 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" podUID="2f96d2a6-501e-4b37-bc62-1f533b3ea517" containerName="payload-processing" containerID="cri-o://d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241" gracePeriod=25 Oct 30 15:37:08.749261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:08.749228 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" podUID="2d1db2b4-f1e7-4464-90bd-5829d090abeb" containerName="payload-pre-processing" containerID="cri-o://afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea" gracePeriod=25 Oct 30 15:37:09.075986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.075947 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:37:09.083609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.083578 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp\" (UniqueName: \"kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp\") pod \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " Oct 30 15:37:09.083787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.083770 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume\") pod \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\" (UID: \"2f96d2a6-501e-4b37-bc62-1f533b3ea517\") " Oct 30 15:37:09.084705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.084635 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume" pod "2f96d2a6-501e-4b37-bc62-1f533b3ea517" (UID: "2f96d2a6-501e-4b37-bc62-1f533b3ea517"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:09.087263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.087228 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp" pod "2f96d2a6-501e-4b37-bc62-1f533b3ea517" (UID: "2f96d2a6-501e-4b37-bc62-1f533b3ea517"). InnerVolumeSpecName "kube-api-access-6c4kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:09.093643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.093619 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:37:09.184894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.184853 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc\" (UniqueName: \"kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc\") pod \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " Oct 30 15:37:09.185088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.184967 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume\") pod \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\" (UID: \"2d1db2b4-f1e7-4464-90bd-5829d090abeb\") " Oct 30 15:37:09.185767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.185743 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume" pod "2d1db2b4-f1e7-4464-90bd-5829d090abeb" (UID: "2d1db2b4-f1e7-4464-90bd-5829d090abeb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:09.185932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.185778 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f96d2a6-501e-4b37-bc62-1f533b3ea517-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.186051 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.186035 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4kp\" (UniqueName: \"kubernetes.io/projected/2f96d2a6-501e-4b37-bc62-1f533b3ea517-kube-api-access-6c4kp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.188166 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.188138 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc" pod "2d1db2b4-f1e7-4464-90bd-5829d090abeb" (UID: "2d1db2b4-f1e7-4464-90bd-5829d090abeb"). InnerVolumeSpecName "kube-api-access-mwphc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:09.210501 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:09.210457 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:37:09.287387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.287329 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwphc\" (UniqueName: \"kubernetes.io/projected/2d1db2b4-f1e7-4464-90bd-5829d090abeb-kube-api-access-mwphc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.287387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.287383 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1db2b4-f1e7-4464-90bd-5829d090abeb-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:09.707246 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.707213 6554 generic.go:356] "Generic (PLEG): container finished" podID="2f96d2a6-501e-4b37-bc62-1f533b3ea517" containerID="d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241" exitCode=0 Oct 30 15:37:09.707461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.707278 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" Oct 30 15:37:09.707461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.707302 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" event={"ID":"2f96d2a6-501e-4b37-bc62-1f533b3ea517","Type":"ContainerDied","Data":"d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241"} Oct 30 15:37:09.707461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.707415 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6" event={"ID":"2f96d2a6-501e-4b37-bc62-1f533b3ea517","Type":"ContainerDied","Data":"0d8be69ac0768fbccb9ba7581f0a484eed40decb70036fe224cfe3dbdde61d0b"} Oct 30 15:37:09.707461 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.707446 6554 scope.go:122] "RemoveContainer" containerID="d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241" Oct 30 15:37:09.708970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.708948 6554 generic.go:356] "Generic (PLEG): container finished" podID="2d1db2b4-f1e7-4464-90bd-5829d090abeb" containerID="afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea" exitCode=0 Oct 30 15:37:09.709049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.708998 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" Oct 30 15:37:09.709095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.709047 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" event={"ID":"2d1db2b4-f1e7-4464-90bd-5829d090abeb","Type":"ContainerDied","Data":"afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea"} Oct 30 15:37:09.709136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.709100 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92" event={"ID":"2d1db2b4-f1e7-4464-90bd-5829d090abeb","Type":"ContainerDied","Data":"a3b73a6c0e261b53064cac6f0f0d20912d8270e87cd87bd681d4cab0d155a302"} Oct 30 15:37:09.719281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.719062 6554 scope.go:122] "RemoveContainer" containerID="d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241" Oct 30 15:37:09.719655 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:09.719412 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241\": container with ID starting with d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241 not found: ID does not exist" containerID="d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241" Oct 30 15:37:09.719655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.719444 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241"} err="failed to get container status \"d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241\": rpc error: code = NotFound desc = could not find container \"d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241\": container with ID starting with d3590cbba7e0b2a6837bd86b154c711fe47b49fb9765ece1df20883ca650a241 not found: ID does not exist" Oct 30 15:37:09.719655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.719465 6554 scope.go:122] "RemoveContainer" containerID="afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea" Oct 30 15:37:09.746297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.746277 6554 scope.go:122] "RemoveContainer" containerID="afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea" Oct 30 15:37:09.746614 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:09.746596 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea\": container with ID starting with afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea not found: ID does not exist" containerID="afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea" Oct 30 15:37:09.746665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.746623 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea"} err="failed to get container status \"afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea\": rpc error: code = NotFound desc = could not find container \"afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea\": container with ID starting with afe77ac6f20886c29cd26e1644d4dab53a35d4142c6075ac54ee1ca2e0eaf7ea not found: ID does not exist" Oct 30 15:37:09.766155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.766134 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92"] Oct 30 15:37:09.781529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.781505 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-908638-7775fdcb55-zwj92"] Oct 30 15:37:09.808997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.808977 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6"] Oct 30 15:37:09.815926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:09.815895 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-908638-559d97f694-p4md6"] Oct 30 15:37:10.164445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.164414 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1db2b4-f1e7-4464-90bd-5829d090abeb" path="/var/lib/kubelet/pods/2d1db2b4-f1e7-4464-90bd-5829d090abeb/volumes" Oct 30 15:37:10.164898 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.164882 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f96d2a6-501e-4b37-bc62-1f533b3ea517" path="/var/lib/kubelet/pods/2f96d2a6-501e-4b37-bc62-1f533b3ea517/volumes" Oct 30 15:37:10.744866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.744838 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9"] Oct 30 15:37:10.745670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745653 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2d1db2b4-f1e7-4464-90bd-5829d090abeb" containerName="payload-pre-processing" Oct 30 15:37:10.745718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745671 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1db2b4-f1e7-4464-90bd-5829d090abeb" containerName="payload-pre-processing" Oct 30 15:37:10.745718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745687 6554 cpu_manager.go:414] "containerMap: removing container" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" Oct 30 15:37:10.745718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745692 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745708 6554 cpu_manager.go:414] "containerMap: removing container" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745730 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745753 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745758 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745771 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745775 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745786 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745790 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745802 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2f96d2a6-501e-4b37-bc62-1f533b3ea517" containerName="payload-processing" Oct 30 15:37:10.745841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745806 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f96d2a6-501e-4b37-bc62-1f533b3ea517" containerName="payload-processing" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745947 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fd71f9-b19f-4a9d-8c80-e2097316161d" containerName="main" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745963 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6b0c106-9b7e-4f7b-9aea-15bf5272a65a" containerName="main" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745974 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745982 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2ec6b9-f6e2-418c-baee-f6dcd25e2889" containerName="istio-proxy" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.745993 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f96d2a6-501e-4b37-bc62-1f533b3ea517" containerName="payload-processing" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.746007 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1db2b4-f1e7-4464-90bd-5829d090abeb" containerName="payload-pre-processing" Oct 30 15:37:10.746304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.746015 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab2baa3-9253-4029-bbdf-6ef9cbd5faf5" containerName="istio-proxy" Oct 30 15:37:10.749818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.749799 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.770230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.770206 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-261ad0-serving-cert\"" Oct 30 15:37:10.789095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.789072 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9"] Oct 30 15:37:10.819386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.817427 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.819386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.817568 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.819386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.817664 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fswpw\" (UniqueName: \"kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.819386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.817728 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.918899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.918872 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.919064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.918935 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fswpw\" (UniqueName: \"kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.919064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.918974 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.919271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.919250 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.920076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.920050 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.921962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.921919 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.922851 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.922809 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.944511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.944452 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fswpw\" (UniqueName: \"kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw\") pod \"maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:10.991410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:10.991386 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 15:37:11.036462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.036425 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume\") pod \"101b3da3-6528-4ec3-9c33-992975d2c884\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " Oct 30 15:37:11.036917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.036885 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s\" (UniqueName: \"kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s\") pod \"101b3da3-6528-4ec3-9c33-992975d2c884\" (UID: \"101b3da3-6528-4ec3-9c33-992975d2c884\") " Oct 30 15:37:11.052391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.038504 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume" pod "101b3da3-6528-4ec3-9c33-992975d2c884" (UID: "101b3da3-6528-4ec3-9c33-992975d2c884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:11.111487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.110688 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s" pod "101b3da3-6528-4ec3-9c33-992975d2c884" (UID: "101b3da3-6528-4ec3-9c33-992975d2c884"). InnerVolumeSpecName "kube-api-access-66d8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:11.130206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.130171 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:11.139304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.139278 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/101b3da3-6528-4ec3-9c33-992975d2c884-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:11.139449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.139307 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66d8s\" (UniqueName: \"kubernetes.io/projected/101b3da3-6528-4ec3-9c33-992975d2c884-kube-api-access-66d8s\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:11.442675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.442646 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9"] Oct 30 15:37:11.443398 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:37:11.443366 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod103a7f0d_b730_4d9b_a0a0_b2d29c64dcba.slice/crio-8aa4f390e58846f451cfb1a530266bbf058401bb5967ffc7118acbaf99288533 WatchSource:0}: Error finding container 8aa4f390e58846f451cfb1a530266bbf058401bb5967ffc7118acbaf99288533: Status 404 returned error can't find the container with id 8aa4f390e58846f451cfb1a530266bbf058401bb5967ffc7118acbaf99288533 Oct 30 15:37:11.727714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.727677 6554 generic.go:356] "Generic (PLEG): container finished" podID="101b3da3-6528-4ec3-9c33-992975d2c884" containerID="bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f" exitCode=0 Oct 30 15:37:11.727923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.727722 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" Oct 30 15:37:11.727923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.727742 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" event={"ID":"101b3da3-6528-4ec3-9c33-992975d2c884","Type":"ContainerDied","Data":"bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f"} Oct 30 15:37:11.727923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.727807 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46" event={"ID":"101b3da3-6528-4ec3-9c33-992975d2c884","Type":"ContainerDied","Data":"4022a0041bba1fe6093881fddf18309de048f6de0231adb3f544f78e3939e8fa"} Oct 30 15:37:11.727923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.727837 6554 scope.go:122] "RemoveContainer" containerID="bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f" Oct 30 15:37:11.729418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.729390 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" event={"ID":"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba","Type":"ContainerStarted","Data":"8aa4f390e58846f451cfb1a530266bbf058401bb5967ffc7118acbaf99288533"} Oct 30 15:37:11.739840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.739824 6554 scope.go:122] "RemoveContainer" containerID="bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f" Oct 30 15:37:11.740101 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:11.740076 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f\": container with ID starting with bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f not found: ID does not exist" containerID="bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f" Oct 30 15:37:11.740176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.740111 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f"} err="failed to get container status \"bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f\": rpc error: code = NotFound desc = could not find container \"bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f\": container with ID starting with bde7305c59a637f0a10607d7408d2197693326df4f693836ecf92aebc384ff7f not found: ID does not exist" Oct 30 15:37:11.785164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.785139 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46"] Oct 30 15:37:11.799547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:11.799520 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-4f5c30-7554b89b74-rhq46"] Oct 30 15:37:12.182381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:12.182328 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" path="/var/lib/kubelet/pods/101b3da3-6528-4ec3-9c33-992975d2c884/volumes" Oct 30 15:37:12.739922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:12.739895 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" event={"ID":"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba","Type":"ContainerStarted","Data":"8e8ffedaf05e6f8215e3167b17ff77d0a95afbf2d47395709887faaa7c08ac81"} Oct 30 15:37:12.740106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:12.740084 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:12.802487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:12.802428 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" podStartSLOduration=2.411399355 podStartE2EDuration="2.80240808s" totalImagesPullingTime="391.008725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:37:12.784109446 +0000 UTC m=+2936.743150850" watchObservedRunningTime="2026-10-30 15:37:12.80240808 +0000 UTC m=+2936.761449489" Oct 30 15:37:18.753800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:18.753774 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:37:34.283272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.283238 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p"] Oct 30 15:37:34.284313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.283483 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" podUID="da273e05-66b8-4ce7-8a5a-8a0ba247c554" containerName="maas-api" containerID="cri-o://81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa" gracePeriod=30 Oct 30 15:37:34.680194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.679831 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:37:34.771955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.771922 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca\") pod \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " Oct 30 15:37:34.772182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.772095 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\") pod \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " Oct 30 15:37:34.772182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.772146 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4\" (UniqueName: \"kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4\") pod \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " Oct 30 15:37:34.772182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.772175 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs\") pod \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\" (UID: \"da273e05-66b8-4ce7-8a5a-8a0ba247c554\") " Oct 30 15:37:34.772368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.772310 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca" pod "da273e05-66b8-4ce7-8a5a-8a0ba247c554" (UID: "da273e05-66b8-4ce7-8a5a-8a0ba247c554"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:34.772968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.772942 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da273e05-66b8-4ce7-8a5a-8a0ba247c554-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:34.776985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.776957 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs" pod "da273e05-66b8-4ce7-8a5a-8a0ba247c554" (UID: "da273e05-66b8-4ce7-8a5a-8a0ba247c554"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:37:34.777143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.776981 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls" pod "da273e05-66b8-4ce7-8a5a-8a0ba247c554" (UID: "da273e05-66b8-4ce7-8a5a-8a0ba247c554"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:37:34.777271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.777246 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4" pod "da273e05-66b8-4ce7-8a5a-8a0ba247c554" (UID: "da273e05-66b8-4ce7-8a5a-8a0ba247c554"). InnerVolumeSpecName "kube-api-access-jqnr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:34.874800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.874775 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:34.874800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.874799 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqnr4\" (UniqueName: \"kubernetes.io/projected/da273e05-66b8-4ce7-8a5a-8a0ba247c554-kube-api-access-jqnr4\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:34.874949 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:34.874811 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da273e05-66b8-4ce7-8a5a-8a0ba247c554-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:35.013830 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.013277 6554 generic.go:356] "Generic (PLEG): container finished" podID="da273e05-66b8-4ce7-8a5a-8a0ba247c554" containerID="81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa" exitCode=0 Oct 30 15:37:35.018093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.014975 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" Oct 30 15:37:35.025959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.025929 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" event={"ID":"da273e05-66b8-4ce7-8a5a-8a0ba247c554","Type":"ContainerDied","Data":"81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa"} Oct 30 15:37:35.026665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.026638 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p" event={"ID":"da273e05-66b8-4ce7-8a5a-8a0ba247c554","Type":"ContainerDied","Data":"912de47a50989213734f01d65861cbbbf258cfa325839c752a644148ce5774ef"} Oct 30 15:37:35.027018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.026990 6554 scope.go:122] "RemoveContainer" containerID="81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa" Oct 30 15:37:35.046956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.046661 6554 scope.go:122] "RemoveContainer" containerID="81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa" Oct 30 15:37:35.048363 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:35.048031 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa\": container with ID starting with 81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa not found: ID does not exist" containerID="81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa" Oct 30 15:37:35.048363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.048074 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa"} err="failed to get container status \"81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa\": rpc error: code = NotFound desc = could not find container \"81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa\": container with ID starting with 81cca6ccca50cc85b2fc779c6d76dbfb6de19f925a83a3165f1946deffc791fa not found: ID does not exist" Oct 30 15:37:35.091241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.091207 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p"] Oct 30 15:37:35.104472 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:35.104445 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-648b1c-7d6cc6959d-2tm8p"] Oct 30 15:37:36.157101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:36.157061 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da273e05-66b8-4ce7-8a5a-8a0ba247c554" path="/var/lib/kubelet/pods/da273e05-66b8-4ce7-8a5a-8a0ba247c554/volumes" Oct 30 15:37:39.208885 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:39.208837 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:37:45.133141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:45.133104 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt"] Oct 30 15:37:45.184767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:45.184729 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2"] Oct 30 15:37:50.134398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.134342 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" podUID="6492473e-58d4-4b90-bd3d-b1fcd157e6cb" containerName="payload-processing" containerID="cri-o://71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e" gracePeriod=25 Oct 30 15:37:50.185828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.185795 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" podUID="b667d5d7-3ff2-4114-b315-184883aa521e" containerName="payload-pre-processing" containerID="cri-o://a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57" gracePeriod=25 Oct 30 15:37:50.522681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.522650 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:37:50.565321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.565299 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:37:50.626084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.626048 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume\") pod \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " Oct 30 15:37:50.626292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.626278 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b\" (UniqueName: \"kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b\") pod \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\" (UID: \"6492473e-58d4-4b90-bd3d-b1fcd157e6cb\") " Oct 30 15:37:50.626532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.626508 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume" pod "6492473e-58d4-4b90-bd3d-b1fcd157e6cb" (UID: "6492473e-58d4-4b90-bd3d-b1fcd157e6cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:50.626987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.626965 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:50.628899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.628876 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b" pod "6492473e-58d4-4b90-bd3d-b1fcd157e6cb" (UID: "6492473e-58d4-4b90-bd3d-b1fcd157e6cb"). InnerVolumeSpecName "kube-api-access-rlj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:50.727619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.727555 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg\" (UniqueName: \"kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg\") pod \"b667d5d7-3ff2-4114-b315-184883aa521e\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " Oct 30 15:37:50.727757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.727634 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume\") pod \"b667d5d7-3ff2-4114-b315-184883aa521e\" (UID: \"b667d5d7-3ff2-4114-b315-184883aa521e\") " Oct 30 15:37:50.727966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.727946 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume" pod "b667d5d7-3ff2-4114-b315-184883aa521e" (UID: "b667d5d7-3ff2-4114-b315-184883aa521e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:37:50.728275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.728255 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlj4b\" (UniqueName: \"kubernetes.io/projected/6492473e-58d4-4b90-bd3d-b1fcd157e6cb-kube-api-access-rlj4b\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:50.728342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.728280 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b667d5d7-3ff2-4114-b315-184883aa521e-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:50.729581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.729562 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg" pod "b667d5d7-3ff2-4114-b315-184883aa521e" (UID: "b667d5d7-3ff2-4114-b315-184883aa521e"). InnerVolumeSpecName "kube-api-access-48hfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:37:50.829856 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:50.829827 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48hfg\" (UniqueName: \"kubernetes.io/projected/b667d5d7-3ff2-4114-b315-184883aa521e-kube-api-access-48hfg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:37:51.163113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.162887 6554 generic.go:356] "Generic (PLEG): container finished" podID="e58e5fda-c858-4072-8631-307319269c5e" containerID="11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c" exitCode=0 Oct 30 15:37:51.163113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.162917 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" event={"ID":"e58e5fda-c858-4072-8631-307319269c5e","Type":"ContainerDied","Data":"11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c"} Oct 30 15:37:51.163646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.163305 6554 scope.go:122] "RemoveContainer" containerID="11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c" Oct 30 15:37:51.165154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165126 6554 generic.go:356] "Generic (PLEG): container finished" podID="b667d5d7-3ff2-4114-b315-184883aa521e" containerID="a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57" exitCode=0 Oct 30 15:37:51.165295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165260 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:37:51.165379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165295 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" event={"ID":"b667d5d7-3ff2-4114-b315-184883aa521e","Type":"ContainerDied","Data":"a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57"} Oct 30 15:37:51.165379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165337 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" event={"ID":"b667d5d7-3ff2-4114-b315-184883aa521e","Type":"ContainerDied","Data":"7aafc4f4ff7cd11b5dba710a82b724e38cb58fc2d1dee8cb7956b8652dfe6ae9"} Oct 30 15:37:51.165521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165376 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:37:51.165521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165281 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2" Oct 30 15:37:51.165521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165428 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:37:51.165521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.165391 6554 scope.go:122] "RemoveContainer" containerID="a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57" Oct 30 15:37:51.179334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.179307 6554 generic.go:356] "Generic (PLEG): container finished" podID="6492473e-58d4-4b90-bd3d-b1fcd157e6cb" containerID="71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e" exitCode=0 Oct 30 15:37:51.179452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.179396 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" event={"ID":"6492473e-58d4-4b90-bd3d-b1fcd157e6cb","Type":"ContainerDied","Data":"71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e"} Oct 30 15:37:51.179514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.179498 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" event={"ID":"6492473e-58d4-4b90-bd3d-b1fcd157e6cb","Type":"ContainerDied","Data":"858e6c7c5783a1e853e99e8a838083c39d27c4ef17ad42b08702cf7a5ccf91c7"} Oct 30 15:37:51.179594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.179573 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt" Oct 30 15:37:51.195905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.195886 6554 scope.go:122] "RemoveContainer" containerID="a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57" Oct 30 15:37:51.196231 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:51.196204 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57\": container with ID starting with a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57 not found: ID does not exist" containerID="a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57" Oct 30 15:37:51.196326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.196240 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57"} err="failed to get container status \"a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57\": rpc error: code = NotFound desc = could not find container \"a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57\": container with ID starting with a0b248a7a79bf8088e3475c26682ed2e2b4ea64d4334e9474a06dfb2e2113c57 not found: ID does not exist" Oct 30 15:37:51.196326 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.196263 6554 scope.go:122] "RemoveContainer" containerID="71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e" Oct 30 15:37:51.208544 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.208515 6554 scope.go:122] "RemoveContainer" containerID="71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e" Oct 30 15:37:51.208829 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:37:51.208807 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e\": container with ID starting with 71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e not found: ID does not exist" containerID="71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e" Oct 30 15:37:51.208913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.208833 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e"} err="failed to get container status \"71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e\": rpc error: code = NotFound desc = could not find container \"71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e\": container with ID starting with 71b832f9699885780d7219ebc6bb39f9a8c4de0943c73c9c80c1db54937c875e not found: ID does not exist" Oct 30 15:37:51.270274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.270242 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt"] Oct 30 15:37:51.285130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.285101 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-0644a3-55b5bd7588-zc8vt"] Oct 30 15:37:51.317458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.317425 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2"] Oct 30 15:37:51.326167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:51.326128 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-0644a3-5d847bdc47-v2rd2"] Oct 30 15:37:52.156525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:52.156491 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6492473e-58d4-4b90-bd3d-b1fcd157e6cb" path="/var/lib/kubelet/pods/6492473e-58d4-4b90-bd3d-b1fcd157e6cb/volumes" Oct 30 15:37:52.156912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:52.156898 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b667d5d7-3ff2-4114-b315-184883aa521e" path="/var/lib/kubelet/pods/b667d5d7-3ff2-4114-b315-184883aa521e/volumes" Oct 30 15:37:52.189666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:52.189640 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" event={"ID":"e58e5fda-c858-4072-8631-307319269c5e","Type":"ContainerStarted","Data":"0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189"} Oct 30 15:37:52.691594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:52.691555 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:37:52.693247 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:52.693219 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:37:52.693420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:52.693270 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 15:37:53.692542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:53.692505 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:37:53.693019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:53.692561 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 15:37:53.901962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:53.901931 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:37:54.692659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:54.692622 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:37:54.693071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:54.692677 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 15:37:55.692678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:55.692631 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:37:55.693221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:55.692720 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 15:37:56.695881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:56.695844 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:37:56.696692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:56.696658 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 15:37:57.692532 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:57.692497 6554 patch_prober.go:29] interesting pod/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body= Oct 30 15:37:57.692718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:57.692559 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" Oct 30 15:37:58.696567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:58.696534 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:37:58.697604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:37:58.697580 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:38:03.936321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:03.936289 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5"] Oct 30 15:38:03.973482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:03.973454 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj"] Oct 30 15:38:05.477038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.477001 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n"] Oct 30 15:38:05.477939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.477899 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" podUID="dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" containerName="maas-api" containerID="cri-o://469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba" gracePeriod=30 Oct 30 15:38:05.864375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.864332 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:38:05.957648 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.957602 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca\") pod \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " Oct 30 15:38:05.957875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.957855 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\") pod \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " Oct 30 15:38:05.957946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.957889 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs\") pod \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " Oct 30 15:38:05.957946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.957931 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca" pod "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" (UID: "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:05.958068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.957951 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc\" (UniqueName: \"kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc\") pod \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\" (UID: \"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d\") " Oct 30 15:38:05.958584 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.958556 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:05.960285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.960260 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc" pod "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" (UID: "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d"). InnerVolumeSpecName "kube-api-access-vtwlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:05.960598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.960571 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs" pod "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" (UID: "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:05.960598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:05.960584 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls" pod "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" (UID: "dcf9f4c3-828f-4c14-8d2f-690f8f100e6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:38:06.060084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.060018 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.060084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.060045 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.060084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.060055 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtwlc\" (UniqueName: \"kubernetes.io/projected/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d-kube-api-access-vtwlc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:06.346591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.346496 6554 generic.go:356] "Generic (PLEG): container finished" podID="dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" containerID="469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba" exitCode=0 Oct 30 15:38:06.346769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.346607 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" event={"ID":"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d","Type":"ContainerDied","Data":"469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba"} Oct 30 15:38:06.346769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.346660 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" event={"ID":"dcf9f4c3-828f-4c14-8d2f-690f8f100e6d","Type":"ContainerDied","Data":"588c993a4a277c3bb22f671fa366a4dbd529fce737c275fa6278aeff164e41d5"} Oct 30 15:38:06.346769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.346692 6554 scope.go:122] "RemoveContainer" containerID="469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba" Oct 30 15:38:06.346947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.346832 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n" Oct 30 15:38:06.359422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.359396 6554 scope.go:122] "RemoveContainer" containerID="469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba" Oct 30 15:38:06.359733 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:06.359711 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba\": container with ID starting with 469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba not found: ID does not exist" containerID="469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba" Oct 30 15:38:06.359786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.359742 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba"} err="failed to get container status \"469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba\": rpc error: code = NotFound desc = could not find container \"469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba\": container with ID starting with 469fc5c652a89e878b310a5e1273dae9741559106b6d4d776fd4b14ea051c8ba not found: ID does not exist" Oct 30 15:38:06.415090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.415057 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n"] Oct 30 15:38:06.430504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:06.430474 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f3a578-9555f5bf5-bgr4n"] Oct 30 15:38:08.154857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:08.154825 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" path="/var/lib/kubelet/pods/dcf9f4c3-828f-4c14-8d2f-690f8f100e6d/volumes" Oct 30 15:38:08.936585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:08.936540 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" podUID="6cb2d6c6-35eb-45de-a356-23f3369c68a5" containerName="payload-processing" containerID="cri-o://400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85" gracePeriod=25 Oct 30 15:38:08.974391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:08.974339 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" podUID="4ccd772a-10fc-48f8-bcc5-27e24af681f4" containerName="payload-pre-processing" containerID="cri-o://bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1" gracePeriod=25 Oct 30 15:38:09.207483 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:09.206230 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:38:09.262152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.262119 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:38:09.296468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.296447 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:38:09.354768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.354743 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume\") pod \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " Oct 30 15:38:09.354930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.354795 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj\" (UniqueName: \"kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj\") pod \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " Oct 30 15:38:09.354930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.354851 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume\") pod \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\" (UID: \"6cb2d6c6-35eb-45de-a356-23f3369c68a5\") " Oct 30 15:38:09.355064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.355009 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd\" (UniqueName: \"kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd\") pod \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\" (UID: \"4ccd772a-10fc-48f8-bcc5-27e24af681f4\") " Oct 30 15:38:09.355242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.355212 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume" pod "4ccd772a-10fc-48f8-bcc5-27e24af681f4" (UID: "4ccd772a-10fc-48f8-bcc5-27e24af681f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:09.355603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.355576 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume" pod "6cb2d6c6-35eb-45de-a356-23f3369c68a5" (UID: "6cb2d6c6-35eb-45de-a356-23f3369c68a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:09.355730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.355713 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb2d6c6-35eb-45de-a356-23f3369c68a5-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:09.355809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.355732 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ccd772a-10fc-48f8-bcc5-27e24af681f4-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:09.357181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.357144 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd" pod "4ccd772a-10fc-48f8-bcc5-27e24af681f4" (UID: "4ccd772a-10fc-48f8-bcc5-27e24af681f4"). InnerVolumeSpecName "kube-api-access-g22xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:09.357282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.357175 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj" pod "6cb2d6c6-35eb-45de-a356-23f3369c68a5" (UID: "6cb2d6c6-35eb-45de-a356-23f3369c68a5"). InnerVolumeSpecName "kube-api-access-wstkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:09.375440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.375408 6554 generic.go:356] "Generic (PLEG): container finished" podID="4ccd772a-10fc-48f8-bcc5-27e24af681f4" containerID="bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1" exitCode=0 Oct 30 15:38:09.375547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.375457 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" Oct 30 15:38:09.375547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.375490 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" event={"ID":"4ccd772a-10fc-48f8-bcc5-27e24af681f4","Type":"ContainerDied","Data":"bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1"} Oct 30 15:38:09.375687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.375559 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj" event={"ID":"4ccd772a-10fc-48f8-bcc5-27e24af681f4","Type":"ContainerDied","Data":"9fee82821cbb5214e637c9a4c0f59bbaae90fa522f320a64fd4b9782cd62bd78"} Oct 30 15:38:09.375687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.375591 6554 scope.go:122] "RemoveContainer" containerID="bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1" Oct 30 15:38:09.377471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.377446 6554 generic.go:356] "Generic (PLEG): container finished" podID="6cb2d6c6-35eb-45de-a356-23f3369c68a5" containerID="400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85" exitCode=0 Oct 30 15:38:09.377471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.377502 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" event={"ID":"6cb2d6c6-35eb-45de-a356-23f3369c68a5","Type":"ContainerDied","Data":"400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85"} Oct 30 15:38:09.377471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.377516 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" Oct 30 15:38:09.377471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.377539 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5" event={"ID":"6cb2d6c6-35eb-45de-a356-23f3369c68a5","Type":"ContainerDied","Data":"4e71a75a5df007db27f9267efb73267d0281dc04b6fc6eccb4074aefee8eeafa"} Oct 30 15:38:09.389958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.389917 6554 scope.go:122] "RemoveContainer" containerID="bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1" Oct 30 15:38:09.390202 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:09.390182 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1\": container with ID starting with bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1 not found: ID does not exist" containerID="bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1" Oct 30 15:38:09.390294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.390212 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1"} err="failed to get container status \"bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1\": rpc error: code = NotFound desc = could not find container \"bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1\": container with ID starting with bed96ca68ed29964d72d05f71f0d168fc0b8091a983ae8fae173ae2ad07863d1 not found: ID does not exist" Oct 30 15:38:09.390294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.390239 6554 scope.go:122] "RemoveContainer" containerID="400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85" Oct 30 15:38:09.402718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.402698 6554 scope.go:122] "RemoveContainer" containerID="400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85" Oct 30 15:38:09.402981 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:09.402960 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85\": container with ID starting with 400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85 not found: ID does not exist" containerID="400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85" Oct 30 15:38:09.403047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.402989 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85"} err="failed to get container status \"400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85\": rpc error: code = NotFound desc = could not find container \"400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85\": container with ID starting with 400b5d0aedbe63b3bf8e150c7b46b86863012000fae90fad133bc1b9db27ab85 not found: ID does not exist" Oct 30 15:38:09.451737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.451713 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj"] Oct 30 15:38:09.455985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.455963 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-e30f62-5f994cbc9f-hzrrj"] Oct 30 15:38:09.456788 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.456769 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstkj\" (UniqueName: \"kubernetes.io/projected/6cb2d6c6-35eb-45de-a356-23f3369c68a5-kube-api-access-wstkj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:09.456846 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.456791 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g22xd\" (UniqueName: \"kubernetes.io/projected/4ccd772a-10fc-48f8-bcc5-27e24af681f4-kube-api-access-g22xd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:09.499333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.499279 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5"] Oct 30 15:38:09.502086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:09.502068 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-e30f62-595cbd49fd-swqd5"] Oct 30 15:38:10.156631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:10.156600 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccd772a-10fc-48f8-bcc5-27e24af681f4" path="/var/lib/kubelet/pods/4ccd772a-10fc-48f8-bcc5-27e24af681f4/volumes" Oct 30 15:38:10.157038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:10.157024 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb2d6c6-35eb-45de-a356-23f3369c68a5" path="/var/lib/kubelet/pods/6cb2d6c6-35eb-45de-a356-23f3369c68a5/volumes" Oct 30 15:38:10.773134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:10.773098 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q"] Oct 30 15:38:10.773632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:10.773343 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" containerID="cri-o://e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5" gracePeriod=30 Oct 30 15:38:10.997591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:10.997544 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:38:10.998201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:10.998168 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:38:16.071089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.071063 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:38:16.156817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.156781 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.156837 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.156871 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.156900 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.156934 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.156977 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.157052 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.157243 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157294 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.157283 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs\") pod \"db381190-c5f6-46c1-a14e-51a4c413aaae\" (UID: \"db381190-c5f6-46c1-a14e-51a4c413aaae\") " Oct 30 15:38:16.157655 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.157626 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.158179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.157873 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.158179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.158052 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.158179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.158099 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:16.158803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.158526 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.158803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.158595 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.162003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.161971 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:16.162783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.162746 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:16.163033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.163008 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "kube-api-access-wnpfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:16.163507 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.163472 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo" pod "db381190-c5f6-46c1-a14e-51a4c413aaae" (UID: "db381190-c5f6-46c1-a14e-51a4c413aaae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:38:16.260200 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260167 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260217 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260235 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db381190-c5f6-46c1-a14e-51a4c413aaae-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260295 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260310 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260326 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnpfs\" (UniqueName: \"kubernetes.io/projected/db381190-c5f6-46c1-a14e-51a4c413aaae-kube-api-access-wnpfs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260340 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.260394 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.260384 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db381190-c5f6-46c1-a14e-51a4c413aaae-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:16.449865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.449834 6554 generic.go:356] "Generic (PLEG): container finished" podID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerID="e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5" exitCode=0 Oct 30 15:38:16.450026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.449885 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" Oct 30 15:38:16.450026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.449919 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" event={"ID":"db381190-c5f6-46c1-a14e-51a4c413aaae","Type":"ContainerDied","Data":"e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5"} Oct 30 15:38:16.450026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.449976 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q" event={"ID":"db381190-c5f6-46c1-a14e-51a4c413aaae","Type":"ContainerDied","Data":"7aac56259f9a873e73bc3d0dedb4014ac60e5178cc6e1cc933044c76b3e017f0"} Oct 30 15:38:16.450026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.450000 6554 scope.go:122] "RemoveContainer" containerID="e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5" Oct 30 15:38:16.460558 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.460527 6554 scope.go:122] "RemoveContainer" containerID="e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5" Oct 30 15:38:16.460819 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:16.460797 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5\": container with ID starting with e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5 not found: ID does not exist" containerID="e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5" Oct 30 15:38:16.460887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.460828 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5"} err="failed to get container status \"e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5\": rpc error: code = NotFound desc = could not find container \"e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5\": container with ID starting with e7f53db6aaa62377dc264e3a9790debc7e4fb8b1353eb03fb9721848162118a5 not found: ID does not exist" Oct 30 15:38:16.509630 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.509593 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q"] Oct 30 15:38:16.513578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:16.513558 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-985bf6-openshift-default-596d6c8c59-6bq9q"] Oct 30 15:38:18.155677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:18.155643 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" path="/var/lib/kubelet/pods/db381190-c5f6-46c1-a14e-51a4c413aaae/volumes" Oct 30 15:38:22.303098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:22.303058 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl"] Oct 30 15:38:22.351879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:22.351851 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f"] Oct 30 15:38:27.303433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:27.303377 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" containerName="payload-processing" containerID="cri-o://fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801" gracePeriod=25 Oct 30 15:38:27.352561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:27.352512 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" containerName="payload-pre-processing" containerID="cri-o://9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f" gracePeriod=25 Oct 30 15:38:33.632195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:33.632160 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv"] Oct 30 15:38:33.632669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:33.632378 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" containerID="cri-o://8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a" gracePeriod=30 Oct 30 15:38:33.855885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:33.855825 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:33.865540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:33.864549 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:38.832088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:38.831738 6554 patch_prober.go:29] interesting pod/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" start-of-body= Oct 30 15:38:38.832088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:38.831806 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.211:15021/healthz/ready\": dial tcp 10.134.0.211:15021: connect: connection refused" Oct 30 15:38:38.975894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:38.975870 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:38:39.111788 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.111760 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.111962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.111860 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.111962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.111894 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.111962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.111933 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.112156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.111965 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.112156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112028 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.112156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112065 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.112156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112106 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.112156 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112149 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs\") pod \"01f70536-e123-4dad-810c-b77ab92b8c04\" (UID: \"01f70536-e123-4dad-810c-b77ab92b8c04\") " Oct 30 15:38:39.112473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112339 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:39.112473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112380 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:39.112636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112609 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:39.112709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112621 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:39.112797 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112769 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:39.112993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112972 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.113069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.112999 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.113069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.113015 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.113069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.113029 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01f70536-e123-4dad-810c-b77ab92b8c04-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.113069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.113043 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.114176 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.114157 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:38:39.114376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.114344 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:39.114604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.114580 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:39.114604 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.114593 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm" pod "01f70536-e123-4dad-810c-b77ab92b8c04" (UID: "01f70536-e123-4dad-810c-b77ab92b8c04"). InnerVolumeSpecName "kube-api-access-wnkzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:39.201717 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:39.201658 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:38:39.215157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.215133 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01f70536-e123-4dad-810c-b77ab92b8c04-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.215304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.215160 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnkzm\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-kube-api-access-wnkzm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.215304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.215176 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01f70536-e123-4dad-810c-b77ab92b8c04-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.215304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.215190 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01f70536-e123-4dad-810c-b77ab92b8c04-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:39.690596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.690565 6554 generic.go:356] "Generic (PLEG): container finished" podID="01f70536-e123-4dad-810c-b77ab92b8c04" containerID="8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a" exitCode=0 Oct 30 15:38:39.690792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.690612 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" Oct 30 15:38:39.690792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.690655 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" event={"ID":"01f70536-e123-4dad-810c-b77ab92b8c04","Type":"ContainerDied","Data":"8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a"} Oct 30 15:38:39.690792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.690708 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv" event={"ID":"01f70536-e123-4dad-810c-b77ab92b8c04","Type":"ContainerDied","Data":"32bba0e2d08ec56c7e921f1dbf79478b7f4fc1b5db889d38fcdf0261bd34477f"} Oct 30 15:38:39.690792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.690729 6554 scope.go:122] "RemoveContainer" containerID="8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a" Oct 30 15:38:39.701686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.701669 6554 scope.go:122] "RemoveContainer" containerID="ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f" Oct 30 15:38:39.710791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.710775 6554 scope.go:122] "RemoveContainer" containerID="8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a" Oct 30 15:38:39.711051 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:39.711032 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a\": container with ID starting with 8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a not found: ID does not exist" containerID="8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a" Oct 30 15:38:39.711129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.711062 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a"} err="failed to get container status \"8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a\": rpc error: code = NotFound desc = could not find container \"8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a\": container with ID starting with 8aebbfb75678770091d8f7d21f0501dbce6c1ff74c4734b0a1a2c9bae762254a not found: ID does not exist" Oct 30 15:38:39.711129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.711089 6554 scope.go:122] "RemoveContainer" containerID="ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f" Oct 30 15:38:39.711340 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:39.711324 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f\": container with ID starting with ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f not found: ID does not exist" containerID="ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f" Oct 30 15:38:39.711420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.711344 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f"} err="failed to get container status \"ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f\": rpc error: code = NotFound desc = could not find container \"ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f\": container with ID starting with ba28d50af83fc434320298a0cce0d1aeb175c806bb027a494c9acd1cb483362f not found: ID does not exist" Oct 30 15:38:39.762823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.762794 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv"] Oct 30 15:38:39.776529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:39.776506 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-78b513-openshift-default-5d97ffc9-8d9xv"] Oct 30 15:38:40.158302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:40.158267 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" path="/var/lib/kubelet/pods/01f70536-e123-4dad-810c-b77ab92b8c04/volumes" Oct 30 15:38:43.849031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:43.848992 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:43.849031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:43.849005 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:38:47.645271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.645247 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:38:47.672767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.672743 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:38:47.729611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.729567 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds\" (UniqueName: \"kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds\") pod \"37205325-d62f-4f8e-9255-57f7a14abc89\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " Oct 30 15:38:47.729806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.729780 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume\") pod \"37205325-d62f-4f8e-9255-57f7a14abc89\" (UID: \"37205325-d62f-4f8e-9255-57f7a14abc89\") " Oct 30 15:38:47.730388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.730339 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume" pod "37205325-d62f-4f8e-9255-57f7a14abc89" (UID: "37205325-d62f-4f8e-9255-57f7a14abc89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:47.730545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.730524 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37205325-d62f-4f8e-9255-57f7a14abc89-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:47.733487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.733452 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds" pod "37205325-d62f-4f8e-9255-57f7a14abc89" (UID: "37205325-d62f-4f8e-9255-57f7a14abc89"). InnerVolumeSpecName "kube-api-access-wqcds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:47.767694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.767667 6554 generic.go:356] "Generic (PLEG): container finished" podID="974af636-43be-47d7-b88c-c4f2b50ab696" containerID="9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f" exitCode=0 Oct 30 15:38:47.767805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.767709 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" Oct 30 15:38:47.767805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.767744 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" event={"ID":"974af636-43be-47d7-b88c-c4f2b50ab696","Type":"ContainerDied","Data":"9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f"} Oct 30 15:38:47.767888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.767805 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f" event={"ID":"974af636-43be-47d7-b88c-c4f2b50ab696","Type":"ContainerDied","Data":"8bdb266e2a063754102bed175193ea79240df9be1cd2d67d9f4289236c08a42a"} Oct 30 15:38:47.767888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.767829 6554 scope.go:122] "RemoveContainer" containerID="9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f" Oct 30 15:38:47.769285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.769266 6554 generic.go:356] "Generic (PLEG): container finished" podID="37205325-d62f-4f8e-9255-57f7a14abc89" containerID="fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801" exitCode=0 Oct 30 15:38:47.769406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.769317 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" Oct 30 15:38:47.769406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.769326 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" event={"ID":"37205325-d62f-4f8e-9255-57f7a14abc89","Type":"ContainerDied","Data":"fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801"} Oct 30 15:38:47.769406 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.769375 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl" event={"ID":"37205325-d62f-4f8e-9255-57f7a14abc89","Type":"ContainerDied","Data":"b813738a1036397660478ddaf23eab58af4183f81e35e3a829ed3ff2f6bc721d"} Oct 30 15:38:47.778433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.778405 6554 scope.go:122] "RemoveContainer" containerID="9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f" Oct 30 15:38:47.780198 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:47.780177 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f\": container with ID starting with 9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f not found: ID does not exist" containerID="9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f" Oct 30 15:38:47.780304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.780207 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f"} err="failed to get container status \"9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f\": rpc error: code = NotFound desc = could not find container \"9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f\": container with ID starting with 9ca3e6a1ac23cfcc0a87742aba3bfe555e60ac602690c4c3a9f79a92270e7e2f not found: ID does not exist" Oct 30 15:38:47.780304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.780229 6554 scope.go:122] "RemoveContainer" containerID="fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801" Oct 30 15:38:47.793826 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.793805 6554 scope.go:122] "RemoveContainer" containerID="fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801" Oct 30 15:38:47.794125 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:38:47.794103 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801\": container with ID starting with fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801 not found: ID does not exist" containerID="fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801" Oct 30 15:38:47.794219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.794134 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801"} err="failed to get container status \"fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801\": rpc error: code = NotFound desc = could not find container \"fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801\": container with ID starting with fb723f91a40095a41d836164d7dcc01e84d9096bb4daf7712f74ed112e80b801 not found: ID does not exist" Oct 30 15:38:47.831823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.831756 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v\" (UniqueName: \"kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v\") pod \"974af636-43be-47d7-b88c-c4f2b50ab696\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " Oct 30 15:38:47.831934 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.831887 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume\") pod \"974af636-43be-47d7-b88c-c4f2b50ab696\" (UID: \"974af636-43be-47d7-b88c-c4f2b50ab696\") " Oct 30 15:38:47.832297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.832268 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume" pod "974af636-43be-47d7-b88c-c4f2b50ab696" (UID: "974af636-43be-47d7-b88c-c4f2b50ab696"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:47.832486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.832469 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974af636-43be-47d7-b88c-c4f2b50ab696-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:47.832486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.832487 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqcds\" (UniqueName: \"kubernetes.io/projected/37205325-d62f-4f8e-9255-57f7a14abc89-kube-api-access-wqcds\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:47.833808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.833787 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v" pod "974af636-43be-47d7-b88c-c4f2b50ab696" (UID: "974af636-43be-47d7-b88c-c4f2b50ab696"). InnerVolumeSpecName "kube-api-access-95h9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:47.862761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.862741 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl"] Oct 30 15:38:47.865543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.865523 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-609709-59fd76667b-mtzxl"] Oct 30 15:38:47.933740 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:47.933718 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95h9v\" (UniqueName: \"kubernetes.io/projected/974af636-43be-47d7-b88c-c4f2b50ab696-kube-api-access-95h9v\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:48.123237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:48.123185 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f"] Oct 30 15:38:48.126214 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:48.126197 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-609709-8589b55575-zvm4f"] Oct 30 15:38:48.156112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:48.156090 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" path="/var/lib/kubelet/pods/37205325-d62f-4f8e-9255-57f7a14abc89/volumes" Oct 30 15:38:48.156515 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:48.156502 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" path="/var/lib/kubelet/pods/974af636-43be-47d7-b88c-c4f2b50ab696/volumes" Oct 30 15:38:49.707226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:49.707193 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs"] Oct 30 15:38:49.707739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:49.707454 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" containerID="cri-o://39c4abd6cd6470e0e8e5b3fd342db87340a47b170eb251bfd3f49604b7911bb0" gracePeriod=30 Oct 30 15:38:54.825576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:54.825542 6554 generic.go:356] "Generic (PLEG): container finished" podID="1f6bf27f-8644-461a-a224-dd833934e377" containerID="39c4abd6cd6470e0e8e5b3fd342db87340a47b170eb251bfd3f49604b7911bb0" exitCode=0 Oct 30 15:38:54.825912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:54.825619 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" event={"ID":"1f6bf27f-8644-461a-a224-dd833934e377","Type":"ContainerDied","Data":"39c4abd6cd6470e0e8e5b3fd342db87340a47b170eb251bfd3f49604b7911bb0"} Oct 30 15:38:54.982229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:54.982167 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:38:55.051467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051437 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051514 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051608 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051680 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051729 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051836 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051862 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.051999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051906 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.051999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.051962 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.052112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.052003 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo\") pod \"1f6bf27f-8644-461a-a224-dd833934e377\" (UID: \"1f6bf27f-8644-461a-a224-dd833934e377\") " Oct 30 15:38:55.052170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.052126 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.052506 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.052444 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:38:55.053043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053006 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.053144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053077 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.053341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053317 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.053341 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053345 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.053535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053385 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.053535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053399 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.053535 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.053412 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f6bf27f-8644-461a-a224-dd833934e377-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.054504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.054476 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:55.055036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.055010 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:38:55.055133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.055053 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:38:55.056366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.056326 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9" pod "1f6bf27f-8644-461a-a224-dd833934e377" (UID: "1f6bf27f-8644-461a-a224-dd833934e377"). InnerVolumeSpecName "kube-api-access-brws9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:38:55.154723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.154694 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brws9\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-kube-api-access-brws9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.154723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.154723 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f6bf27f-8644-461a-a224-dd833934e377-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.154866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.154737 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f6bf27f-8644-461a-a224-dd833934e377-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.154866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.154748 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f6bf27f-8644-461a-a224-dd833934e377-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:38:55.231718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.231687 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml"] Oct 30 15:38:55.834469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.834435 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" Oct 30 15:38:55.834959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.834440 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs" event={"ID":"1f6bf27f-8644-461a-a224-dd833934e377","Type":"ContainerDied","Data":"22d22fc1ae784e701a2c9001bc6ee0fcb4165d39aba57cc5d76565910b525166"} Oct 30 15:38:55.834959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.834605 6554 scope.go:122] "RemoveContainer" containerID="39c4abd6cd6470e0e8e5b3fd342db87340a47b170eb251bfd3f49604b7911bb0" Oct 30 15:38:55.911452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.911406 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs"] Oct 30 15:38:55.920091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:55.917979 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-609709-openshift-default-659bb4db79-7psfs"] Oct 30 15:38:56.156623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:56.156594 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f6bf27f-8644-461a-a224-dd833934e377" path="/var/lib/kubelet/pods/1f6bf27f-8644-461a-a224-dd833934e377/volumes" Oct 30 15:38:58.024894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:58.024853 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7"] Oct 30 15:38:58.025462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:38:58.025088 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" containerID="cri-o://7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9" gracePeriod=30 Oct 30 15:39:00.565509 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.565478 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f"] Oct 30 15:39:00.566147 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566132 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6492473e-58d4-4b90-bd3d-b1fcd157e6cb" containerName="payload-processing" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566148 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6492473e-58d4-4b90-bd3d-b1fcd157e6cb" containerName="payload-processing" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566170 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6cb2d6c6-35eb-45de-a356-23f3369c68a5" containerName="payload-processing" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566175 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb2d6c6-35eb-45de-a356-23f3369c68a5" containerName="payload-processing" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566185 6554 cpu_manager.go:414] "containerMap: removing container" podUID="dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" containerName="maas-api" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566190 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" containerName="maas-api" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566200 6554 cpu_manager.go:414] "containerMap: removing container" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" containerName="payload-processing" Oct 30 15:39:00.566211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566205 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" containerName="payload-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566219 6554 cpu_manager.go:414] "containerMap: removing container" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566225 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566232 6554 cpu_manager.go:414] "containerMap: removing container" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566237 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566243 6554 cpu_manager.go:414] "containerMap: removing container" podUID="4ccd772a-10fc-48f8-bcc5-27e24af681f4" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566247 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccd772a-10fc-48f8-bcc5-27e24af681f4" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566257 6554 cpu_manager.go:414] "containerMap: removing container" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" containerName="payload-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566262 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" containerName="payload-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566273 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b667d5d7-3ff2-4114-b315-184883aa521e" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566277 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b667d5d7-3ff2-4114-b315-184883aa521e" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566286 6554 cpu_manager.go:414] "containerMap: removing container" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566290 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566299 6554 cpu_manager.go:414] "containerMap: removing container" podUID="da273e05-66b8-4ce7-8a5a-8a0ba247c554" containerName="maas-api" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566303 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da273e05-66b8-4ce7-8a5a-8a0ba247c554" containerName="maas-api" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566314 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566318 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566328 6554 cpu_manager.go:414] "containerMap: removing container" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566334 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566436 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf9f4c3-828f-4c14-8d2f-690f8f100e6d" containerName="maas-api" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566445 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="101b3da3-6528-4ec3-9c33-992975d2c884" containerName="payload-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566453 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566459 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="974af636-43be-47d7-b88c-c4f2b50ab696" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566465 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccd772a-10fc-48f8-bcc5-27e24af681f4" containerName="payload-pre-processing" Oct 30 15:39:00.566464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566472 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="37205325-d62f-4f8e-9255-57f7a14abc89" containerName="payload-processing" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566478 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b667d5d7-3ff2-4114-b315-184883aa521e" containerName="payload-pre-processing" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566483 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f70536-e123-4dad-810c-b77ab92b8c04" containerName="istio-proxy" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566488 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="da273e05-66b8-4ce7-8a5a-8a0ba247c554" containerName="maas-api" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566495 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f6bf27f-8644-461a-a224-dd833934e377" containerName="istio-proxy" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566501 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="db381190-c5f6-46c1-a14e-51a4c413aaae" containerName="istio-proxy" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566506 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6492473e-58d4-4b90-bd3d-b1fcd157e6cb" containerName="payload-processing" Oct 30 15:39:00.567318 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.566514 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb2d6c6-35eb-45de-a356-23f3369c68a5" containerName="payload-processing" Oct 30 15:39:00.581554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.581527 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.591054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.591031 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f"] Oct 30 15:39:00.599404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.599374 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-01c89d\"" Oct 30 15:39:00.600206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.600184 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-01c89d-dockercfg-6h5hl\"" Oct 30 15:39:00.706373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.696283 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.706373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.696487 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.759414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.756878 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg"] Oct 30 15:39:00.785456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.783874 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg"] Oct 30 15:39:00.785456 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.784078 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:00.799132 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.799096 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.799319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.799303 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.800645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.800241 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.824908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.824860 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl\") pod \"payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.897720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.897695 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:00.900268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.900242 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:00.900448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:00.900426 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nslt8\" (UniqueName: \"kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8\") pod \"payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.003398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.002474 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.003398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.002630 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nslt8\" (UniqueName: \"kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8\") pod \"payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.003575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.003414 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.029727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.029701 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nslt8\" (UniqueName: \"kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8\") pod \"payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.110635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.110562 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.213647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.212760 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f"] Oct 30 15:39:01.227698 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:39:01.227666 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65b66cd_636f_4919_9eaa_d6b8404ef5d4.slice/crio-45a1a995eaf6dd4262b58e0628883c9a1c5b7b33db2eb4470ace91f6f95f2764 WatchSource:0}: Error finding container 45a1a995eaf6dd4262b58e0628883c9a1c5b7b33db2eb4470ace91f6f95f2764: Status 404 returned error can't find the container with id 45a1a995eaf6dd4262b58e0628883c9a1c5b7b33db2eb4470ace91f6f95f2764 Oct 30 15:39:01.458029 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:39:01.457993 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b8f84b_141a_451a_801e_5f7bed08dd95.slice/crio-05fd1e425c14a0502c3156ec5dc914cd8e81c116bc92da301908dc71a051f797 WatchSource:0}: Error finding container 05fd1e425c14a0502c3156ec5dc914cd8e81c116bc92da301908dc71a051f797: Status 404 returned error can't find the container with id 05fd1e425c14a0502c3156ec5dc914cd8e81c116bc92da301908dc71a051f797 Oct 30 15:39:01.458888 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.458861 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg"] Oct 30 15:39:01.894446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.894404 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" event={"ID":"f4b8f84b-141a-451a-801e-5f7bed08dd95","Type":"ContainerStarted","Data":"67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33"} Oct 30 15:39:01.894906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.894470 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" event={"ID":"f4b8f84b-141a-451a-801e-5f7bed08dd95","Type":"ContainerStarted","Data":"05fd1e425c14a0502c3156ec5dc914cd8e81c116bc92da301908dc71a051f797"} Oct 30 15:39:01.894906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.894498 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:01.896026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.896000 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" event={"ID":"f65b66cd-636f-4919-9eaa-d6b8404ef5d4","Type":"ContainerStarted","Data":"1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51"} Oct 30 15:39:01.896103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.896054 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:01.896103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.896072 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" event={"ID":"f65b66cd-636f-4919-9eaa-d6b8404ef5d4","Type":"ContainerStarted","Data":"45a1a995eaf6dd4262b58e0628883c9a1c5b7b33db2eb4470ace91f6f95f2764"} Oct 30 15:39:01.970036 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:01.969993 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" podStartSLOduration=1.969981982 podStartE2EDuration="1.969981982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:01.968057665 +0000 UTC m=+3045.927099070" watchObservedRunningTime="2026-10-30 15:39:01.969981982 +0000 UTC m=+3045.929023411" Oct 30 15:39:02.018786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:02.018742 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" podStartSLOduration=2.018730058 podStartE2EDuration="2.018730058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:02.017206749 +0000 UTC m=+3045.976248158" watchObservedRunningTime="2026-10-30 15:39:02.018730058 +0000 UTC m=+3045.977771463" Oct 30 15:39:03.303154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.303097 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:39:03.382002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.381965 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382021 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382150 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382284 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382192 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382343 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382431 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382461 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382501 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382532 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data\") pod \"83390336-5acc-4b7e-961f-ddb81298a3ba\" (UID: \"83390336-5acc-4b7e-961f-ddb81298a3ba\") " Oct 30 15:39:03.382978 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.382953 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.383343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.383318 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.383589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.383566 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.383726 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.383702 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:03.383838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.383742 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.384815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.384787 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:03.385381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.385318 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:03.386193 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.386169 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "kube-api-access-spbvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:03.386665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.386642 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token" pod "83390336-5acc-4b7e-961f-ddb81298a3ba" (UID: "83390336-5acc-4b7e-961f-ddb81298a3ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:03.484212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484185 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spbvq\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-kube-api-access-spbvq\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484210 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484219 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484228 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484241 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83390336-5acc-4b7e-961f-ddb81298a3ba-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484250 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484258 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484266 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.484442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.484274 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83390336-5acc-4b7e-961f-ddb81298a3ba-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:03.911124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.911087 6554 generic.go:356] "Generic (PLEG): container finished" podID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerID="7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9" exitCode=0 Oct 30 15:39:03.911307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.911143 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" Oct 30 15:39:03.911307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.911143 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" event={"ID":"83390336-5acc-4b7e-961f-ddb81298a3ba","Type":"ContainerDied","Data":"7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9"} Oct 30 15:39:03.911307 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.911282 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7" event={"ID":"83390336-5acc-4b7e-961f-ddb81298a3ba","Type":"ContainerDied","Data":"1f6ac489c4e6d8ed7bd0e152d872ec6ede0104c3534a33a46ceb213c07bf147c"} Oct 30 15:39:03.911469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.911315 6554 scope.go:122] "RemoveContainer" containerID="7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9" Oct 30 15:39:03.922343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.922327 6554 scope.go:122] "RemoveContainer" containerID="7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9" Oct 30 15:39:03.922615 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:03.922596 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9\": container with ID starting with 7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9 not found: ID does not exist" containerID="7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9" Oct 30 15:39:03.922660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.922623 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9"} err="failed to get container status \"7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9\": rpc error: code = NotFound desc = could not find container \"7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9\": container with ID starting with 7f45fa4def556e63132b3ba8678b381b4e4ac007854c5a29e44fb74e6563cfb9 not found: ID does not exist" Oct 30 15:39:03.969181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.969161 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7"] Oct 30 15:39:03.983653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:03.983629 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f3a578-openshift-default-74886c479f-rv2h7"] Oct 30 15:39:04.181769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:04.181672 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" path="/var/lib/kubelet/pods/83390336-5acc-4b7e-961f-ddb81298a3ba/volumes" Oct 30 15:39:04.869924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:04.869886 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d"] Oct 30 15:39:06.741886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:06.741854 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn"] Oct 30 15:39:06.742337 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:06.742043 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" podUID="7efe748d-06f9-4d43-8de8-241ae8b2f04e" containerName="maas-api" containerID="cri-o://186eef5e49d7aeabb5b2290e145d70d5de470bb108b4d5ba25fa685b8b171c78" gracePeriod=30 Oct 30 15:39:06.940316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:06.940276 6554 generic.go:356] "Generic (PLEG): container finished" podID="7efe748d-06f9-4d43-8de8-241ae8b2f04e" containerID="186eef5e49d7aeabb5b2290e145d70d5de470bb108b4d5ba25fa685b8b171c78" exitCode=0 Oct 30 15:39:06.940487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:06.940466 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" event={"ID":"7efe748d-06f9-4d43-8de8-241ae8b2f04e","Type":"ContainerDied","Data":"186eef5e49d7aeabb5b2290e145d70d5de470bb108b4d5ba25fa685b8b171c78"} Oct 30 15:39:07.066196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.064653 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:39:07.157070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.157041 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca\") pod \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " Oct 30 15:39:07.157251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.157143 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls\") pod \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " Oct 30 15:39:07.157251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.157220 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs\") pod \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " Oct 30 15:39:07.157251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.157247 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz\" (UniqueName: \"kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz\") pod \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\" (UID: \"7efe748d-06f9-4d43-8de8-241ae8b2f04e\") " Oct 30 15:39:07.157541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.157471 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca" pod "7efe748d-06f9-4d43-8de8-241ae8b2f04e" (UID: "7efe748d-06f9-4d43-8de8-241ae8b2f04e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:07.157943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.157915 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7efe748d-06f9-4d43-8de8-241ae8b2f04e-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:07.160006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.159977 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz" pod "7efe748d-06f9-4d43-8de8-241ae8b2f04e" (UID: "7efe748d-06f9-4d43-8de8-241ae8b2f04e"). InnerVolumeSpecName "kube-api-access-7mcfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:07.160130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.160028 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs" pod "7efe748d-06f9-4d43-8de8-241ae8b2f04e" (UID: "7efe748d-06f9-4d43-8de8-241ae8b2f04e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:07.160130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.160048 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls" pod "7efe748d-06f9-4d43-8de8-241ae8b2f04e" (UID: "7efe748d-06f9-4d43-8de8-241ae8b2f04e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:07.263950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.260137 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:07.263950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.260175 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7efe748d-06f9-4d43-8de8-241ae8b2f04e-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:07.263950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.260188 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mcfz\" (UniqueName: \"kubernetes.io/projected/7efe748d-06f9-4d43-8de8-241ae8b2f04e-kube-api-access-7mcfz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:07.948969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.948939 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" Oct 30 15:39:07.949466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.948978 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn" event={"ID":"7efe748d-06f9-4d43-8de8-241ae8b2f04e","Type":"ContainerDied","Data":"333af12429f3db0c542f5960b93d8f29f1cfdf434a6f190e01766f130468f294"} Oct 30 15:39:07.949466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:07.949038 6554 scope.go:122] "RemoveContainer" containerID="186eef5e49d7aeabb5b2290e145d70d5de470bb108b4d5ba25fa685b8b171c78" Oct 30 15:39:08.012890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:08.012858 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn"] Oct 30 15:39:08.026764 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:08.026746 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-09e8eb-6fd56b8d8b-x8bpn"] Oct 30 15:39:08.155262 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:08.155233 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efe748d-06f9-4d43-8de8-241ae8b2f04e" path="/var/lib/kubelet/pods/7efe748d-06f9-4d43-8de8-241ae8b2f04e/volumes" Oct 30 15:39:09.202608 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:09.202569 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:39:09.871206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:09.871169 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerName="payload-processing" containerID="cri-o://43ff541d8225bc24b5eeddf2d515cf2ad4044dc295b37ff8284cbcb5a9d97e7d" gracePeriod=25 Oct 30 15:39:10.252633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.252594 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" containerID="cri-o://a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8" gracePeriod=45 Oct 30 15:39:10.573645 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.573619 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:39:10.604569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604541 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm\") pod \"a62ac25a-76ca-4fee-9999-180c82bb8122\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " Oct 30 15:39:10.604679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604589 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home\") pod \"a62ac25a-76ca-4fee-9999-180c82bb8122\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " Oct 30 15:39:10.604679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604668 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir\") pod \"a62ac25a-76ca-4fee-9999-180c82bb8122\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " Oct 30 15:39:10.604767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604740 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs\" (UniqueName: \"kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs\") pod \"a62ac25a-76ca-4fee-9999-180c82bb8122\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " Oct 30 15:39:10.604811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604771 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache\") pod \"a62ac25a-76ca-4fee-9999-180c82bb8122\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " Oct 30 15:39:10.604863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604806 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp\" (UniqueName: \"kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp\") pod \"a62ac25a-76ca-4fee-9999-180c82bb8122\" (UID: \"a62ac25a-76ca-4fee-9999-180c82bb8122\") " Oct 30 15:39:10.604927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604851 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home" pod "a62ac25a-76ca-4fee-9999-180c82bb8122" (UID: "a62ac25a-76ca-4fee-9999-180c82bb8122"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.604990 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.604967 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir" pod "a62ac25a-76ca-4fee-9999-180c82bb8122" (UID: "a62ac25a-76ca-4fee-9999-180c82bb8122"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.605093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.605066 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache" pod "a62ac25a-76ca-4fee-9999-180c82bb8122" (UID: "a62ac25a-76ca-4fee-9999-180c82bb8122"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.605312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.605296 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.605419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.605330 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.605419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.605340 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.606840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.606818 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm" pod "a62ac25a-76ca-4fee-9999-180c82bb8122" (UID: "a62ac25a-76ca-4fee-9999-180c82bb8122"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:10.606931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.606907 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp" pod "a62ac25a-76ca-4fee-9999-180c82bb8122" (UID: "a62ac25a-76ca-4fee-9999-180c82bb8122"). InnerVolumeSpecName "kube-api-access-7czlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:10.606974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.606963 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs" pod "a62ac25a-76ca-4fee-9999-180c82bb8122" (UID: "a62ac25a-76ca-4fee-9999-180c82bb8122"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:10.707111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.707082 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a62ac25a-76ca-4fee-9999-180c82bb8122-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.707111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.707109 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czlp\" (UniqueName: \"kubernetes.io/projected/a62ac25a-76ca-4fee-9999-180c82bb8122-kube-api-access-7czlp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.707302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.707120 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a62ac25a-76ca-4fee-9999-180c82bb8122-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:10.983491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.983458 6554 generic.go:356] "Generic (PLEG): container finished" podID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerID="a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8" exitCode=0 Oct 30 15:39:10.983678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.983507 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" Oct 30 15:39:10.983678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.983534 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" event={"ID":"a62ac25a-76ca-4fee-9999-180c82bb8122","Type":"ContainerDied","Data":"a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8"} Oct 30 15:39:10.983678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.983577 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" event={"ID":"a62ac25a-76ca-4fee-9999-180c82bb8122","Type":"ContainerDied","Data":"2398b702a7038b98b003982d3863f8aa1dddcd1dd29bdcac6e183123c4a3ed1f"} Oct 30 15:39:10.983678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.983598 6554 scope.go:122] "RemoveContainer" containerID="a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8" Oct 30 15:39:10.994148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.994132 6554 scope.go:122] "RemoveContainer" containerID="a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8" Oct 30 15:39:10.994441 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:10.994423 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8\": container with ID starting with a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8 not found: ID does not exist" containerID="a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8" Oct 30 15:39:10.994498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:10.994451 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8"} err="failed to get container status \"a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8\": rpc error: code = NotFound desc = could not find container \"a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8\": container with ID starting with a6d377ad2bce4dc9fd9db771ee514aae478dd5b6ffff934b2e4e52b8135520b8 not found: ID does not exist" Oct 30 15:39:11.049673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:11.049648 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml"] Oct 30 15:39:11.066057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:11.066033 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml"] Oct 30 15:39:11.552443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:11.552400 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-05d396/premium-w0-05d396-kserve-8684968f8f-x5jml" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:39:12.155362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:12.155327 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" path="/var/lib/kubelet/pods/a62ac25a-76ca-4fee-9999-180c82bb8122/volumes" Oct 30 15:39:12.553860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:12.553781 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:12.906103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:12.906074 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:39:12.906311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:12.906244 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:39:16.173141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:16.173109 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k"] Oct 30 15:39:16.173628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:16.173289 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" containerID="cri-o://fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5" gracePeriod=30 Oct 30 15:39:19.429494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:19.429456 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4"] Oct 30 15:39:19.430431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:19.429722 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" containerID="cri-o://48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e" gracePeriod=30 Oct 30 15:39:21.465960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.465934 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:39:21.568087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568059 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568093 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568162 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568218 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568255 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568295 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568422 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568469 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568489 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:21.568705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568609 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket\") pod \"4909339b-d70f-4224-9b44-3dbcefd347ca\" (UID: \"4909339b-d70f-4224-9b44-3dbcefd347ca\") " Oct 30 15:39:21.568945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568703 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:21.568945 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.568912 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:21.569068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569015 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:21.569185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569155 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:21.569496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569468 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.569602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569502 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.569602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569518 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4909339b-d70f-4224-9b44-3dbcefd347ca-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.569602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569530 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.569602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.569542 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.571681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.571640 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:21.572831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.572807 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "kube-api-access-pptf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:21.573202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.573173 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:21.573760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.573733 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token" pod "4909339b-d70f-4224-9b44-3dbcefd347ca" (UID: "4909339b-d70f-4224-9b44-3dbcefd347ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:21.670892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.670867 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.670892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.670890 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.671038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.670900 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4909339b-d70f-4224-9b44-3dbcefd347ca-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:21.671038 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:21.670909 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptf5\" (UniqueName: \"kubernetes.io/projected/4909339b-d70f-4224-9b44-3dbcefd347ca-kube-api-access-pptf5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:22.061371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.061309 6554 generic.go:356] "Generic (PLEG): container finished" podID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerID="fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5" exitCode=0 Oct 30 15:39:22.061549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.061390 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" Oct 30 15:39:22.061549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.061404 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" event={"ID":"4909339b-d70f-4224-9b44-3dbcefd347ca","Type":"ContainerDied","Data":"fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5"} Oct 30 15:39:22.061549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.061460 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k" event={"ID":"4909339b-d70f-4224-9b44-3dbcefd347ca","Type":"ContainerDied","Data":"bb3068e123f68e67fb2cab0fe8aaec114141509586df8dc06eee028f097ec050"} Oct 30 15:39:22.061549 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.061482 6554 scope.go:122] "RemoveContainer" containerID="fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5" Oct 30 15:39:22.073853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.073834 6554 scope.go:122] "RemoveContainer" containerID="fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5" Oct 30 15:39:22.074122 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:22.074097 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5\": container with ID starting with fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5 not found: ID does not exist" containerID="fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5" Oct 30 15:39:22.074175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.074130 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5"} err="failed to get container status \"fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5\": rpc error: code = NotFound desc = could not find container \"fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5\": container with ID starting with fd34c39711cd20eb9c11bc4cfce573917051ce8c89c4cdf017e03907ee698fb5 not found: ID does not exist" Oct 30 15:39:22.133255 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.133232 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k"] Oct 30 15:39:22.137589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.137560 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-09e8eb-openshift-default-567596c5d9-n4v7k"] Oct 30 15:39:22.154815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.154788 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" path="/var/lib/kubelet/pods/4909339b-d70f-4224-9b44-3dbcefd347ca/volumes" Oct 30 15:39:22.554231 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:22.554198 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:24.701618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.701549 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:39:24.808696 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.808660 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.808969 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809098 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809153 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809189 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809097 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:24.809371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809269 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:24.809371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809304 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809413 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809458 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809516 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo\") pod \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\" (UID: \"de262476-c55d-4eb9-84dc-07a4eedbd1b8\") " Oct 30 15:39:24.809715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.809663 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:24.810596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.810143 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.810752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.810737 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.810878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.810866 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.811003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.810800 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:24.811158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.810844 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:24.813882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.813858 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "kube-api-access-lz8gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:24.817324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.817300 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:24.817552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.817510 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:24.817871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.817839 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy" pod "de262476-c55d-4eb9-84dc-07a4eedbd1b8" (UID: "de262476-c55d-4eb9-84dc-07a4eedbd1b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:24.912124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.912094 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.912124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.912122 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.912306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.912132 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz8gw\" (UniqueName: \"kubernetes.io/projected/de262476-c55d-4eb9-84dc-07a4eedbd1b8-kube-api-access-lz8gw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.912306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.912143 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.912306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.912151 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:24.912306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:24.912161 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de262476-c55d-4eb9-84dc-07a4eedbd1b8-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:25.083202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.083133 6554 generic.go:356] "Generic (PLEG): container finished" podID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerID="48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e" exitCode=0 Oct 30 15:39:25.083202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.083184 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" Oct 30 15:39:25.083447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.083225 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" event={"ID":"de262476-c55d-4eb9-84dc-07a4eedbd1b8","Type":"ContainerDied","Data":"48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e"} Oct 30 15:39:25.083447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.083290 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4" event={"ID":"de262476-c55d-4eb9-84dc-07a4eedbd1b8","Type":"ContainerDied","Data":"b4643e9c79c8bedf9213a960687fbd7f7729485ddad66bd5f56abff26319a03f"} Oct 30 15:39:25.083447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.083321 6554 scope.go:122] "RemoveContainer" containerID="48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e" Oct 30 15:39:25.094628 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.094611 6554 scope.go:122] "RemoveContainer" containerID="48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e" Oct 30 15:39:25.094877 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:25.094858 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e\": container with ID starting with 48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e not found: ID does not exist" containerID="48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e" Oct 30 15:39:25.094931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.094883 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e"} err="failed to get container status \"48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e\": rpc error: code = NotFound desc = could not find container \"48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e\": container with ID starting with 48a0d1052ee22251515fff88d9776a8c846afc40127c2a6eb7cc57bc367c4e8e not found: ID does not exist" Oct 30 15:39:25.142775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.142749 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4"] Oct 30 15:39:25.156286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:25.156267 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-631d84-openshift-default-644b5768d5-ggrc4"] Oct 30 15:39:26.156295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:26.156230 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" path="/var/lib/kubelet/pods/de262476-c55d-4eb9-84dc-07a4eedbd1b8/volumes" Oct 30 15:39:27.132167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:27.132087 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7"] Oct 30 15:39:27.170905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:27.170875 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg"] Oct 30 15:39:30.112001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.109536 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq"] Oct 30 15:39:30.112001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.110130 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" podUID="c1a333af-1683-4f75-860f-db7128c29fb0" containerName="maas-api" containerID="cri-o://b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1" gracePeriod=30 Oct 30 15:39:30.146329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.146295 6554 generic.go:356] "Generic (PLEG): container finished" podID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerID="43ff541d8225bc24b5eeddf2d515cf2ad4044dc295b37ff8284cbcb5a9d97e7d" exitCode=0 Oct 30 15:39:30.146493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.146345 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" event={"ID":"ce158e2f-0d04-49f3-a407-5ad3f3602383","Type":"ContainerDied","Data":"43ff541d8225bc24b5eeddf2d515cf2ad4044dc295b37ff8284cbcb5a9d97e7d"} Oct 30 15:39:30.260398 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.260360 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:39:30.298305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.297196 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume\") pod \"ce158e2f-0d04-49f3-a407-5ad3f3602383\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " Oct 30 15:39:30.298305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.297313 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj\" (UniqueName: \"kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj\") pod \"ce158e2f-0d04-49f3-a407-5ad3f3602383\" (UID: \"ce158e2f-0d04-49f3-a407-5ad3f3602383\") " Oct 30 15:39:30.298305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.297660 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume" pod "ce158e2f-0d04-49f3-a407-5ad3f3602383" (UID: "ce158e2f-0d04-49f3-a407-5ad3f3602383"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:30.298305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.297971 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce158e2f-0d04-49f3-a407-5ad3f3602383-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.323721 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.323671 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj" pod "ce158e2f-0d04-49f3-a407-5ad3f3602383" (UID: "ce158e2f-0d04-49f3-a407-5ad3f3602383"). InnerVolumeSpecName "kube-api-access-xllcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:30.389417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.389377 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:39:30.399129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.399091 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xllcj\" (UniqueName: \"kubernetes.io/projected/ce158e2f-0d04-49f3-a407-5ad3f3602383-kube-api-access-xllcj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.512451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.512414 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs\") pod \"c1a333af-1683-4f75-860f-db7128c29fb0\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " Oct 30 15:39:30.512650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.512591 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\") pod \"c1a333af-1683-4f75-860f-db7128c29fb0\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " Oct 30 15:39:30.512727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.512704 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca\") pod \"c1a333af-1683-4f75-860f-db7128c29fb0\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " Oct 30 15:39:30.512786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.512770 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8\" (UniqueName: \"kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8\") pod \"c1a333af-1683-4f75-860f-db7128c29fb0\" (UID: \"c1a333af-1683-4f75-860f-db7128c29fb0\") " Oct 30 15:39:30.513298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.513270 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca" pod "c1a333af-1683-4f75-860f-db7128c29fb0" (UID: "c1a333af-1683-4f75-860f-db7128c29fb0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:30.514463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.514326 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a333af-1683-4f75-860f-db7128c29fb0-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.516060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.516028 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw"] Oct 30 15:39:30.516386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.516333 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs" pod "c1a333af-1683-4f75-860f-db7128c29fb0" (UID: "c1a333af-1683-4f75-860f-db7128c29fb0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:30.517736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.517687 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls" pod "c1a333af-1683-4f75-860f-db7128c29fb0" (UID: "c1a333af-1683-4f75-860f-db7128c29fb0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:30.519271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.519247 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8" pod "c1a333af-1683-4f75-860f-db7128c29fb0" (UID: "c1a333af-1683-4f75-860f-db7128c29fb0"). InnerVolumeSpecName "kube-api-access-rstp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:30.570765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.570734 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2"] Oct 30 15:39:30.616411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.616382 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.616565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.616414 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstp8\" (UniqueName: \"kubernetes.io/projected/c1a333af-1683-4f75-860f-db7128c29fb0-kube-api-access-rstp8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:30.616565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:30.616430 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1a333af-1683-4f75-860f-db7128c29fb0-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:31.155096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.155021 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" Oct 30 15:39:31.155096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.155032 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d" event={"ID":"ce158e2f-0d04-49f3-a407-5ad3f3602383","Type":"ContainerDied","Data":"f6046da02056ac175d1729929713e53653a6f4415e7ce4debaafd163f892412a"} Oct 30 15:39:31.155096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.155094 6554 scope.go:122] "RemoveContainer" containerID="43ff541d8225bc24b5eeddf2d515cf2ad4044dc295b37ff8284cbcb5a9d97e7d" Oct 30 15:39:31.156739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.156704 6554 generic.go:356] "Generic (PLEG): container finished" podID="c1a333af-1683-4f75-860f-db7128c29fb0" containerID="b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1" exitCode=0 Oct 30 15:39:31.156862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.156747 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" Oct 30 15:39:31.156862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.156785 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" event={"ID":"c1a333af-1683-4f75-860f-db7128c29fb0","Type":"ContainerDied","Data":"b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1"} Oct 30 15:39:31.156862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.156841 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq" event={"ID":"c1a333af-1683-4f75-860f-db7128c29fb0","Type":"ContainerDied","Data":"db14cfcdfb586e16e246702fae15b803451a678fd029ddf0ad052e8f84423163"} Oct 30 15:39:31.167635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.167552 6554 scope.go:122] "RemoveContainer" containerID="b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1" Oct 30 15:39:31.177368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.177337 6554 scope.go:122] "RemoveContainer" containerID="b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1" Oct 30 15:39:31.177656 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:31.177637 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1\": container with ID starting with b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1 not found: ID does not exist" containerID="b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1" Oct 30 15:39:31.177708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.177664 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1"} err="failed to get container status \"b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1\": rpc error: code = NotFound desc = could not find container \"b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1\": container with ID starting with b85d27163d62ee67c8eb0d44a400316acc5c36fa5769fd009495c4fd32672fb1 not found: ID does not exist" Oct 30 15:39:31.223159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.223137 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq"] Oct 30 15:39:31.236642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.236623 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-51bd24-5b9dc7855-fstxq"] Oct 30 15:39:31.289723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.287410 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d"] Oct 30 15:39:31.298409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:31.298376 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-09e8eb-6cd988cd54-7pg6d"] Oct 30 15:39:32.133055 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:32.133015 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerName="payload-processing" containerID="cri-o://a86380c8fe4c87ff48a3f3764e798c8e32e988869f5db32f84df12438ff6957c" gracePeriod=25 Oct 30 15:39:32.154799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:32.154767 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a333af-1683-4f75-860f-db7128c29fb0" path="/var/lib/kubelet/pods/c1a333af-1683-4f75-860f-db7128c29fb0/volumes" Oct 30 15:39:32.155165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:32.155149 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" path="/var/lib/kubelet/pods/ce158e2f-0d04-49f3-a407-5ad3f3602383/volumes" Oct 30 15:39:32.171642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:32.171612 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" podUID="73043333-745a-4a8a-a431-6f4b493d1600" containerName="payload-pre-processing" containerID="cri-o://26fa22b98700e80f16984a18548f88db3cd309f4c9d8ff0d3572c5df38757202" gracePeriod=25 Oct 30 15:39:35.517382 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:35.517325 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" podUID="60e63662-dd11-41ba-be9c-48db54705e50" containerName="payload-processing" containerID="cri-o://a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085" gracePeriod=25 Oct 30 15:39:35.573702 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:35.572927 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" podUID="ac9864e8-ab83-406e-b412-f88427a3671d" containerName="payload-pre-processing" containerID="cri-o://51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d" gracePeriod=25 Oct 30 15:39:35.882841 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:35.882810 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:39:35.937603 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:35.937575 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:39:36.028078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.028052 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258\" (UniqueName: \"kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258\") pod \"60e63662-dd11-41ba-be9c-48db54705e50\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " Oct 30 15:39:36.028221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.028118 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk\" (UniqueName: \"kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk\") pod \"ac9864e8-ab83-406e-b412-f88427a3671d\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " Oct 30 15:39:36.028221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.028212 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume\") pod \"ac9864e8-ab83-406e-b412-f88427a3671d\" (UID: \"ac9864e8-ab83-406e-b412-f88427a3671d\") " Oct 30 15:39:36.028304 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.028234 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume\") pod \"60e63662-dd11-41ba-be9c-48db54705e50\" (UID: \"60e63662-dd11-41ba-be9c-48db54705e50\") " Oct 30 15:39:36.028676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.028615 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume" pod "ac9864e8-ab83-406e-b412-f88427a3671d" (UID: "ac9864e8-ab83-406e-b412-f88427a3671d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:36.028676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.028636 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume" pod "60e63662-dd11-41ba-be9c-48db54705e50" (UID: "60e63662-dd11-41ba-be9c-48db54705e50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:36.030129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.030111 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258" pod "60e63662-dd11-41ba-be9c-48db54705e50" (UID: "60e63662-dd11-41ba-be9c-48db54705e50"). InnerVolumeSpecName "kube-api-access-4z258". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:36.030213 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.030201 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk" pod "ac9864e8-ab83-406e-b412-f88427a3671d" (UID: "ac9864e8-ab83-406e-b412-f88427a3671d"). InnerVolumeSpecName "kube-api-access-lqcdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:36.129653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.129625 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z258\" (UniqueName: \"kubernetes.io/projected/60e63662-dd11-41ba-be9c-48db54705e50-kube-api-access-4z258\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:36.129653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.129651 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqcdk\" (UniqueName: \"kubernetes.io/projected/ac9864e8-ab83-406e-b412-f88427a3671d-kube-api-access-lqcdk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:36.129819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.129662 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac9864e8-ab83-406e-b412-f88427a3671d-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:36.129819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.129671 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60e63662-dd11-41ba-be9c-48db54705e50-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:36.203122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.203087 6554 generic.go:356] "Generic (PLEG): container finished" podID="60e63662-dd11-41ba-be9c-48db54705e50" containerID="a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085" exitCode=0 Oct 30 15:39:36.203260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.203135 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" Oct 30 15:39:36.203260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.203164 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" event={"ID":"60e63662-dd11-41ba-be9c-48db54705e50","Type":"ContainerDied","Data":"a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085"} Oct 30 15:39:36.203260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.203218 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw" event={"ID":"60e63662-dd11-41ba-be9c-48db54705e50","Type":"ContainerDied","Data":"ef0106738cd7b2c924d6e51304766f8a32897609423f6cf761795f5adb69bbc9"} Oct 30 15:39:36.203260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.203249 6554 scope.go:122] "RemoveContainer" containerID="a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085" Oct 30 15:39:36.204592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.204569 6554 generic.go:356] "Generic (PLEG): container finished" podID="ac9864e8-ab83-406e-b412-f88427a3671d" containerID="51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d" exitCode=0 Oct 30 15:39:36.204678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.204612 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" Oct 30 15:39:36.204678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.204657 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" event={"ID":"ac9864e8-ab83-406e-b412-f88427a3671d","Type":"ContainerDied","Data":"51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d"} Oct 30 15:39:36.204776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.204713 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2" event={"ID":"ac9864e8-ab83-406e-b412-f88427a3671d","Type":"ContainerDied","Data":"788bd7ff6bfdcb91828ba57bd5adbf7426c55210d970670c6bc3e59966bff96c"} Oct 30 15:39:36.212905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.212888 6554 scope.go:122] "RemoveContainer" containerID="a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085" Oct 30 15:39:36.213193 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:36.213169 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085\": container with ID starting with a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085 not found: ID does not exist" containerID="a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085" Oct 30 15:39:36.213244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.213205 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085"} err="failed to get container status \"a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085\": rpc error: code = NotFound desc = could not find container \"a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085\": container with ID starting with a63b7ca826220dad2170a450c80f5405aca51fe5dbbb55eb3804110fd7448085 not found: ID does not exist" Oct 30 15:39:36.213244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.213221 6554 scope.go:122] "RemoveContainer" containerID="51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d" Oct 30 15:39:36.223640 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.223609 6554 scope.go:122] "RemoveContainer" containerID="51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d" Oct 30 15:39:36.223895 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:36.223874 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d\": container with ID starting with 51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d not found: ID does not exist" containerID="51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d" Oct 30 15:39:36.223957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.223903 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d"} err="failed to get container status \"51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d\": rpc error: code = NotFound desc = could not find container \"51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d\": container with ID starting with 51f87391fe87be2cbd2729f32c671b4a58b8a94d001d0ed193e0bbba60291c3d not found: ID does not exist" Oct 30 15:39:36.268813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.268789 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2"] Oct 30 15:39:36.280022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.279970 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d57911-8ff98c4cf-qxgj2"] Oct 30 15:39:36.322979 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.322954 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw"] Oct 30 15:39:36.327469 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:36.327449 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d57911-6cf85d9d64-rmwlw"] Oct 30 15:39:37.561621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:37.561579 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" podUID="73043333-745a-4a8a-a431-6f4b493d1600" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:37.562003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:37.561671 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:38.130336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.130211 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l"] Oct 30 15:39:38.130595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.130446 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" podUID="6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" containerName="maas-api" containerID="cri-o://d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda" gracePeriod=30 Oct 30 15:39:38.162476 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.162450 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e63662-dd11-41ba-be9c-48db54705e50" path="/var/lib/kubelet/pods/60e63662-dd11-41ba-be9c-48db54705e50/volumes" Oct 30 15:39:38.162914 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.162895 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac9864e8-ab83-406e-b412-f88427a3671d" path="/var/lib/kubelet/pods/ac9864e8-ab83-406e-b412-f88427a3671d/volumes" Oct 30 15:39:38.397139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.397107 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:39:38.558517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.558473 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca\") pod \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " Oct 30 15:39:38.558718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.558633 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs\") pod \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " Oct 30 15:39:38.558793 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.558774 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls\") pod \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " Oct 30 15:39:38.558848 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.558822 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca" pod "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" (UID: "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:38.558923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.558871 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h\" (UniqueName: \"kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h\") pod \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\" (UID: \"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471\") " Oct 30 15:39:38.559317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.559294 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:38.561057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.561029 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls" pod "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" (UID: "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:38.561151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.561138 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h" pod "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" (UID: "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471"). InnerVolumeSpecName "kube-api-access-7t94h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:38.561261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.561243 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs" pod "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" (UID: "6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:38.668660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.661686 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:38.668660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.661727 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:38.668660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.661741 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t94h\" (UniqueName: \"kubernetes.io/projected/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471-kube-api-access-7t94h\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:38.940909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.940837 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt"] Oct 30 15:39:38.941485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941471 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ac9864e8-ab83-406e-b412-f88427a3671d" containerName="payload-pre-processing" Oct 30 15:39:38.941533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941486 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac9864e8-ab83-406e-b412-f88427a3671d" containerName="payload-pre-processing" Oct 30 15:39:38.941533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941497 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" Oct 30 15:39:38.941533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941503 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" Oct 30 15:39:38.941533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941523 6554 cpu_manager.go:414] "containerMap: removing container" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" Oct 30 15:39:38.941533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941530 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941548 6554 cpu_manager.go:414] "containerMap: removing container" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941553 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941560 6554 cpu_manager.go:414] "containerMap: removing container" podUID="60e63662-dd11-41ba-be9c-48db54705e50" containerName="payload-processing" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941565 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e63662-dd11-41ba-be9c-48db54705e50" containerName="payload-processing" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941571 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerName="payload-processing" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941575 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerName="payload-processing" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941594 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7efe748d-06f9-4d43-8de8-241ae8b2f04e" containerName="maas-api" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941598 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efe748d-06f9-4d43-8de8-241ae8b2f04e" containerName="maas-api" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941605 6554 cpu_manager.go:414] "containerMap: removing container" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941618 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941634 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c1a333af-1683-4f75-860f-db7128c29fb0" containerName="maas-api" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941642 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a333af-1683-4f75-860f-db7128c29fb0" containerName="maas-api" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941683 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" containerName="maas-api" Oct 30 15:39:38.941824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941693 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" containerName="maas-api" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941852 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a333af-1683-4f75-860f-db7128c29fb0" containerName="maas-api" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941867 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" containerName="maas-api" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941879 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac9864e8-ab83-406e-b412-f88427a3671d" containerName="payload-pre-processing" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941887 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a62ac25a-76ca-4fee-9999-180c82bb8122" containerName="main" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941898 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="de262476-c55d-4eb9-84dc-07a4eedbd1b8" containerName="istio-proxy" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941916 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="4909339b-d70f-4224-9b44-3dbcefd347ca" containerName="istio-proxy" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941926 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e63662-dd11-41ba-be9c-48db54705e50" containerName="payload-processing" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941936 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce158e2f-0d04-49f3-a407-5ad3f3602383" containerName="payload-processing" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941947 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efe748d-06f9-4d43-8de8-241ae8b2f04e" containerName="maas-api" Oct 30 15:39:38.942339 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.941954 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="83390336-5acc-4b7e-961f-ddb81298a3ba" containerName="istio-proxy" Oct 30 15:39:38.945561 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.945546 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:38.964111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.964072 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:38.964223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.964131 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:38.964302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.964281 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:38.964369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.964341 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8t2n\" (UniqueName: \"kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:38.979730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.979708 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-01c89d-serving-cert\"" Oct 30 15:39:38.998817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:38.998787 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt"] Oct 30 15:39:39.065694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.065666 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8t2n\" (UniqueName: \"kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.065874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.065726 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.065874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.065779 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.065995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.065949 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.066098 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:39.066078 6554 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-serving-cert: secret "maas-api-e2e-ipp-b-01c89d-serving-cert" not found Oct 30 15:39:39.066152 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:39.066141 6554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls podName:236fbc06-3de3-4f3b-9da8-9de010521aa2 nodeName:}" failed. No retries permitted until 2026-10-30 15:39:39.566123249 +0000 UTC m=+3083.525164636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls") pod "maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" (UID: "236fbc06-3de3-4f3b-9da8-9de010521aa2") : secret "maas-api-e2e-ipp-b-01c89d-serving-cert" not found Oct 30 15:39:39.066559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.066538 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.068206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.068183 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.093334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.093315 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8t2n\" (UniqueName: \"kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.257314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.256917 6554 generic.go:356] "Generic (PLEG): container finished" podID="6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" containerID="d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda" exitCode=0 Oct 30 15:39:39.257314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.257044 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" event={"ID":"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471","Type":"ContainerDied","Data":"d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda"} Oct 30 15:39:39.257314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.257093 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" event={"ID":"6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471","Type":"ContainerDied","Data":"03c3f1950b6733530e30c5dce68b2fdbe3bcb7be6e925764fc79a82e0b2a6fb7"} Oct 30 15:39:39.257314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.257123 6554 scope.go:122] "RemoveContainer" containerID="d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda" Oct 30 15:39:39.257314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.257227 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l" Oct 30 15:39:39.287623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.287593 6554 scope.go:122] "RemoveContainer" containerID="d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda" Oct 30 15:39:39.288566 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:39.288481 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda\": container with ID starting with d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda not found: ID does not exist" containerID="d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda" Oct 30 15:39:39.288566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.288525 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda"} err="failed to get container status \"d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda\": rpc error: code = NotFound desc = could not find container \"d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda\": container with ID starting with d47a1440cb5c437ad4607b921d01c50e8ec92cddaddd01ae449dcb7260bc0cda not found: ID does not exist" Oct 30 15:39:39.290043 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:39.289964 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:39:39.329658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.329632 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l"] Oct 30 15:39:39.333785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.333760 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-d57911-574b9987f9-6g78l"] Oct 30 15:39:39.573903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.573807 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.576293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.576263 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:39.859551 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:39.859488 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:40.154922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:40.154894 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471" path="/var/lib/kubelet/pods/6a1b51cc-fb11-4ccc-81d4-3a0de4d8d471/volumes" Oct 30 15:39:40.164828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:40.164800 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt"] Oct 30 15:39:40.165333 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:39:40.165309 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod236fbc06_3de3_4f3b_9da8_9de010521aa2.slice/crio-66043a7d6659f614f696a5d87d3294d1180388fe28982e68c9d1f37213b00bd5 WatchSource:0}: Error finding container 66043a7d6659f614f696a5d87d3294d1180388fe28982e68c9d1f37213b00bd5: Status 404 returned error can't find the container with id 66043a7d6659f614f696a5d87d3294d1180388fe28982e68c9d1f37213b00bd5 Oct 30 15:39:40.166681 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:40.166663 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:39:40.265291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:40.265264 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" event={"ID":"236fbc06-3de3-4f3b-9da8-9de010521aa2","Type":"ContainerStarted","Data":"66043a7d6659f614f696a5d87d3294d1180388fe28982e68c9d1f37213b00bd5"} Oct 30 15:39:41.273023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:41.272978 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" event={"ID":"236fbc06-3de3-4f3b-9da8-9de010521aa2","Type":"ContainerStarted","Data":"7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798"} Oct 30 15:39:41.273585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:41.273192 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:41.321143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:41.321099 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" podStartSLOduration=2.853386325 podStartE2EDuration="3.321086344s" totalImagesPullingTime="467.700019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:39:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:39:41.318583278 +0000 UTC m=+3085.277624683" watchObservedRunningTime="2026-10-30 15:39:41.321086344 +0000 UTC m=+3085.280127793" Oct 30 15:39:46.026960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:46.026920 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2"] Oct 30 15:39:46.027533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:46.027163 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" containerID="cri-o://ef1b495ef88c1815d41a016573f129e177b144aa9393496f4bd24a503dd32f02" gracePeriod=30 Oct 30 15:39:47.303783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:47.303754 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:39:47.562139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:47.562042 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" podUID="73043333-745a-4a8a-a431-6f4b493d1600" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:47.562139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:47.562082 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:39:50.769597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:50.769564 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:39:50.770079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:50.769627 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:39:51.356416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.356305 6554 generic.go:356] "Generic (PLEG): container finished" podID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerID="ef1b495ef88c1815d41a016573f129e177b144aa9393496f4bd24a503dd32f02" exitCode=0 Oct 30 15:39:51.356416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.356377 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" event={"ID":"7e0098b7-a192-4820-9aad-026cb459eb4a","Type":"ContainerDied","Data":"ef1b495ef88c1815d41a016573f129e177b144aa9393496f4bd24a503dd32f02"} Oct 30 15:39:51.702683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.702660 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:39:51.848317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848292 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848393 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848517 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848561 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848622 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848651 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848681 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848734 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.848754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848741 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:51.849278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848779 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert\") pod \"7e0098b7-a192-4820-9aad-026cb459eb4a\" (UID: \"7e0098b7-a192-4820-9aad-026cb459eb4a\") " Oct 30 15:39:51.849278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848843 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:51.849278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.848877 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:51.849278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849047 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:51.849513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849274 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:51.849513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849475 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.849513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849497 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.849513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849511 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.849683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849526 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.849683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.849540 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0098b7-a192-4820-9aad-026cb459eb4a-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.850884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.850858 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "kube-api-access-z256c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:51.851002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.850954 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:51.851002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.850980 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:39:51.851098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.851084 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo" pod "7e0098b7-a192-4820-9aad-026cb459eb4a" (UID: "7e0098b7-a192-4820-9aad-026cb459eb4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:39:51.950623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.950598 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.950623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.950622 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.950773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.950632 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z256c\" (UniqueName: \"kubernetes.io/projected/7e0098b7-a192-4820-9aad-026cb459eb4a-kube-api-access-z256c\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:51.950773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:51.950642 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0098b7-a192-4820-9aad-026cb459eb4a-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:52.365619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.365579 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" event={"ID":"7e0098b7-a192-4820-9aad-026cb459eb4a","Type":"ContainerDied","Data":"f7b79dcafeb95719e40b2ae5606ae07b1aff51bfd0c0920a6e9de6640be61f33"} Oct 30 15:39:52.365799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.365655 6554 scope.go:122] "RemoveContainer" containerID="ef1b495ef88c1815d41a016573f129e177b144aa9393496f4bd24a503dd32f02" Oct 30 15:39:52.365876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.365820 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2" Oct 30 15:39:52.371249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.371211 6554 generic.go:356] "Generic (PLEG): container finished" podID="73043333-745a-4a8a-a431-6f4b493d1600" containerID="26fa22b98700e80f16984a18548f88db3cd309f4c9d8ff0d3572c5df38757202" exitCode=0 Oct 30 15:39:52.371442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.371422 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" event={"ID":"73043333-745a-4a8a-a431-6f4b493d1600","Type":"ContainerDied","Data":"26fa22b98700e80f16984a18548f88db3cd309f4c9d8ff0d3572c5df38757202"} Oct 30 15:39:52.377980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.377955 6554 generic.go:356] "Generic (PLEG): container finished" podID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerID="a86380c8fe4c87ff48a3f3764e798c8e32e988869f5db32f84df12438ff6957c" exitCode=0 Oct 30 15:39:52.378127 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.378042 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" event={"ID":"dfd8f9b8-62f3-4055-836f-6808267ab03d","Type":"ContainerDied","Data":"a86380c8fe4c87ff48a3f3764e798c8e32e988869f5db32f84df12438ff6957c"} Oct 30 15:39:52.441460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.441431 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2"] Oct 30 15:39:52.454058 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.454030 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-d57911-openshift-default-75489d7cdc-ng2v2"] Oct 30 15:39:52.454980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.454963 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:39:52.475646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.475624 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:39:52.557969 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.557941 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c\" (UniqueName: \"kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c\") pod \"dfd8f9b8-62f3-4055-836f-6808267ab03d\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " Oct 30 15:39:52.558121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.558046 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume\") pod \"73043333-745a-4a8a-a431-6f4b493d1600\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " Oct 30 15:39:52.558224 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.558210 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume\") pod \"dfd8f9b8-62f3-4055-836f-6808267ab03d\" (UID: \"dfd8f9b8-62f3-4055-836f-6808267ab03d\") " Oct 30 15:39:52.558317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.558304 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n\" (UniqueName: \"kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n\") pod \"73043333-745a-4a8a-a431-6f4b493d1600\" (UID: \"73043333-745a-4a8a-a431-6f4b493d1600\") " Oct 30 15:39:52.558614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.558583 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume" pod "73043333-745a-4a8a-a431-6f4b493d1600" (UID: "73043333-745a-4a8a-a431-6f4b493d1600"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:52.558734 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.558617 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume" pod "dfd8f9b8-62f3-4055-836f-6808267ab03d" (UID: "dfd8f9b8-62f3-4055-836f-6808267ab03d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:52.559118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.559098 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfd8f9b8-62f3-4055-836f-6808267ab03d-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:52.559202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.559120 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73043333-745a-4a8a-a431-6f4b493d1600-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:52.560375 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.560338 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n" pod "73043333-745a-4a8a-a431-6f4b493d1600" (UID: "73043333-745a-4a8a-a431-6f4b493d1600"). InnerVolumeSpecName "kube-api-access-z284n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:52.560651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.560638 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c" pod "dfd8f9b8-62f3-4055-836f-6808267ab03d" (UID: "dfd8f9b8-62f3-4055-836f-6808267ab03d"). InnerVolumeSpecName "kube-api-access-bt77c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:52.660464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.660430 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z284n\" (UniqueName: \"kubernetes.io/projected/73043333-745a-4a8a-a431-6f4b493d1600-kube-api-access-z284n\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:52.660464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:52.660457 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt77c\" (UniqueName: \"kubernetes.io/projected/dfd8f9b8-62f3-4055-836f-6808267ab03d-kube-api-access-bt77c\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:53.385553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.385522 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" Oct 30 15:39:53.386002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.385548 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7" event={"ID":"dfd8f9b8-62f3-4055-836f-6808267ab03d","Type":"ContainerDied","Data":"fe5869d7c53ede34b3a236480cc86beba82cb4a4489bebafb53ddee8e1badc0a"} Oct 30 15:39:53.386002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.385624 6554 scope.go:122] "RemoveContainer" containerID="a86380c8fe4c87ff48a3f3764e798c8e32e988869f5db32f84df12438ff6957c" Oct 30 15:39:53.388331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.388304 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" event={"ID":"73043333-745a-4a8a-a431-6f4b493d1600","Type":"ContainerDied","Data":"87ad4fe724cc13814205bd1cf3a1ddee3395e149344e94b99e9284b7ece8d629"} Oct 30 15:39:53.388455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.388316 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg" Oct 30 15:39:53.398896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.398877 6554 scope.go:122] "RemoveContainer" containerID="26fa22b98700e80f16984a18548f88db3cd309f4c9d8ff0d3572c5df38757202" Oct 30 15:39:53.455895 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.455867 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg"] Oct 30 15:39:53.469006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.468984 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-97ead4-788b4856c-557pg"] Oct 30 15:39:53.503512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.503490 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7"] Oct 30 15:39:53.507735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:53.507716 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-97ead4-647ffdf698-2dcn7"] Oct 30 15:39:54.154982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.154954 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73043333-745a-4a8a-a431-6f4b493d1600" path="/var/lib/kubelet/pods/73043333-745a-4a8a-a431-6f4b493d1600/volumes" Oct 30 15:39:54.155467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.155448 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" path="/var/lib/kubelet/pods/7e0098b7-a192-4820-9aad-026cb459eb4a/volumes" Oct 30 15:39:54.155825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.155812 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" path="/var/lib/kubelet/pods/dfd8f9b8-62f3-4055-836f-6808267ab03d/volumes" Oct 30 15:39:54.419641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.418486 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl"] Oct 30 15:39:54.419641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.418722 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" podUID="b5ea4576-a5ed-494c-a471-eea268e6548a" containerName="maas-api" containerID="cri-o://ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d" gracePeriod=30 Oct 30 15:39:54.745963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.745939 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:39:54.889300 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.889263 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls\") pod \"b5ea4576-a5ed-494c-a471-eea268e6548a\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " Oct 30 15:39:54.889533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.889383 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs\") pod \"b5ea4576-a5ed-494c-a471-eea268e6548a\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " Oct 30 15:39:54.889533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.889430 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca\") pod \"b5ea4576-a5ed-494c-a471-eea268e6548a\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " Oct 30 15:39:54.889533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.889466 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57\" (UniqueName: \"kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57\") pod \"b5ea4576-a5ed-494c-a471-eea268e6548a\" (UID: \"b5ea4576-a5ed-494c-a471-eea268e6548a\") " Oct 30 15:39:54.889860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.889834 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca" pod "b5ea4576-a5ed-494c-a471-eea268e6548a" (UID: "b5ea4576-a5ed-494c-a471-eea268e6548a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:39:54.890211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.890190 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5ea4576-a5ed-494c-a471-eea268e6548a-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:54.891801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.891771 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls" pod "b5ea4576-a5ed-494c-a471-eea268e6548a" (UID: "b5ea4576-a5ed-494c-a471-eea268e6548a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:54.892091 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.892071 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs" pod "b5ea4576-a5ed-494c-a471-eea268e6548a" (UID: "b5ea4576-a5ed-494c-a471-eea268e6548a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:39:54.892290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.892265 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57" pod "b5ea4576-a5ed-494c-a471-eea268e6548a" (UID: "b5ea4576-a5ed-494c-a471-eea268e6548a"). InnerVolumeSpecName "kube-api-access-2tl57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:39:54.991763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.991702 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:54.991763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.991728 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5ea4576-a5ed-494c-a471-eea268e6548a-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:54.991763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:54.991738 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tl57\" (UniqueName: \"kubernetes.io/projected/b5ea4576-a5ed-494c-a471-eea268e6548a-kube-api-access-2tl57\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:39:55.404937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.404903 6554 generic.go:356] "Generic (PLEG): container finished" podID="b5ea4576-a5ed-494c-a471-eea268e6548a" containerID="ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d" exitCode=0 Oct 30 15:39:55.405122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.404966 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" Oct 30 15:39:55.405122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.404988 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" event={"ID":"b5ea4576-a5ed-494c-a471-eea268e6548a","Type":"ContainerDied","Data":"ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d"} Oct 30 15:39:55.405122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.405063 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl" event={"ID":"b5ea4576-a5ed-494c-a471-eea268e6548a","Type":"ContainerDied","Data":"409b57a1ecdda2b8fb222106e10adc71de49cf7fc4abf8cacaee7a8d8d469d93"} Oct 30 15:39:55.405122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.405086 6554 scope.go:122] "RemoveContainer" containerID="ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d" Oct 30 15:39:55.415919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.415905 6554 scope.go:122] "RemoveContainer" containerID="ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d" Oct 30 15:39:55.416173 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:39:55.416153 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d\": container with ID starting with ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d not found: ID does not exist" containerID="ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d" Oct 30 15:39:55.416230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.416181 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d"} err="failed to get container status \"ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d\": rpc error: code = NotFound desc = could not find container \"ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d\": container with ID starting with ec15bc06bde64cc3570e61b7231b6cf7d64c7f164fc58372e4fe675b48e16b7d not found: ID does not exist" Oct 30 15:39:55.462663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.462642 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl"] Oct 30 15:39:55.475947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:55.475925 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-97ead4-d7ddd4cc6-g67kl"] Oct 30 15:39:56.155812 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:56.155781 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5ea4576-a5ed-494c-a471-eea268e6548a" path="/var/lib/kubelet/pods/b5ea4576-a5ed-494c-a471-eea268e6548a/volumes" Oct 30 15:39:57.059386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:39:57.059340 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg"] Oct 30 15:40:02.060589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.060533 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" podUID="4afb49d4-4ab0-45c2-8fdc-190326fbcb39" containerName="payload-processing" containerID="cri-o://6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866" gracePeriod=25 Oct 30 15:40:02.341400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.341374 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:40:02.467524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.467487 6554 generic.go:356] "Generic (PLEG): container finished" podID="4afb49d4-4ab0-45c2-8fdc-190326fbcb39" containerID="6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866" exitCode=0 Oct 30 15:40:02.467672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.467537 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" Oct 30 15:40:02.467672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.467540 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" event={"ID":"4afb49d4-4ab0-45c2-8fdc-190326fbcb39","Type":"ContainerDied","Data":"6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866"} Oct 30 15:40:02.467672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.467659 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg" event={"ID":"4afb49d4-4ab0-45c2-8fdc-190326fbcb39","Type":"ContainerDied","Data":"33c4e53ef9c6e82aaa82ad92f56f7f0c39d96ff3faeacbd3d5a5d63838dfe380"} Oct 30 15:40:02.467795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.467687 6554 scope.go:122] "RemoveContainer" containerID="6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866" Oct 30 15:40:02.477202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.477180 6554 scope.go:122] "RemoveContainer" containerID="6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866" Oct 30 15:40:02.477479 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:02.477461 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866\": container with ID starting with 6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866 not found: ID does not exist" containerID="6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866" Oct 30 15:40:02.477567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.477486 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866"} err="failed to get container status \"6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866\": rpc error: code = NotFound desc = could not find container \"6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866\": container with ID starting with 6886ce08e00263583e0983d7931072e56453c23ff2a0925f19cbee4f93888866 not found: ID does not exist" Oct 30 15:40:02.477808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.477789 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume\") pod \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " Oct 30 15:40:02.477886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.477868 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm\" (UniqueName: \"kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm\") pod \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\" (UID: \"4afb49d4-4ab0-45c2-8fdc-190326fbcb39\") " Oct 30 15:40:02.478119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.478102 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume" pod "4afb49d4-4ab0-45c2-8fdc-190326fbcb39" (UID: "4afb49d4-4ab0-45c2-8fdc-190326fbcb39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:02.478287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.478276 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:02.480143 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.480120 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm" pod "4afb49d4-4ab0-45c2-8fdc-190326fbcb39" (UID: "4afb49d4-4ab0-45c2-8fdc-190326fbcb39"). InnerVolumeSpecName "kube-api-access-xtwrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:02.579828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.579776 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwrm\" (UniqueName: \"kubernetes.io/projected/4afb49d4-4ab0-45c2-8fdc-190326fbcb39-kube-api-access-xtwrm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:02.817492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.817457 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg"] Oct 30 15:40:02.820635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:02.820608 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-76473c-89fcdc4d-rw7fg"] Oct 30 15:40:04.154763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:04.154730 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4afb49d4-4ab0-45c2-8fdc-190326fbcb39" path="/var/lib/kubelet/pods/4afb49d4-4ab0-45c2-8fdc-190326fbcb39/volumes" Oct 30 15:40:08.482746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:08.482706 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4"] Oct 30 15:40:08.515449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:08.515418 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl"] Oct 30 15:40:08.524753 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:08.524729 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh"] Oct 30 15:40:09.201157 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:09.201119 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:40:14.132238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.132202 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z"] Oct 30 15:40:14.361879 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.361840 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9"] Oct 30 15:40:14.362113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.362087 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" podUID="103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" containerName="maas-api" containerID="cri-o://8e8ffedaf05e6f8215e3167b17ff77d0a95afbf2d47395709887faaa7c08ac81" gracePeriod=30 Oct 30 15:40:14.561842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.561810 6554 generic.go:356] "Generic (PLEG): container finished" podID="103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" containerID="8e8ffedaf05e6f8215e3167b17ff77d0a95afbf2d47395709887faaa7c08ac81" exitCode=0 Oct 30 15:40:14.562003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.561936 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" event={"ID":"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba","Type":"ContainerDied","Data":"8e8ffedaf05e6f8215e3167b17ff77d0a95afbf2d47395709887faaa7c08ac81"} Oct 30 15:40:14.687002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.686974 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:40:14.760553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.760521 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca\") pod \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " Oct 30 15:40:14.760728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.760563 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw\" (UniqueName: \"kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw\") pod \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " Oct 30 15:40:14.760728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.760587 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs\") pod \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " Oct 30 15:40:14.760728 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.760642 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls\") pod \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\" (UID: \"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba\") " Oct 30 15:40:14.760926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.760867 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca" pod "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" (UID: "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:14.761180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.761160 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:14.762931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.762906 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls" pod "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" (UID: "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:14.763254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.763219 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs" pod "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" (UID: "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:14.763381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.763291 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw" pod "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" (UID: "103a7f0d-b730-4d9b-a0a0-b2d29c64dcba"). InnerVolumeSpecName "kube-api-access-fswpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:14.862180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.862155 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fswpw\" (UniqueName: \"kubernetes.io/projected/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-kube-api-access-fswpw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:14.862274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.862181 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:14.862274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:14.862191 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:15.583957 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.583919 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" event={"ID":"103a7f0d-b730-4d9b-a0a0-b2d29c64dcba","Type":"ContainerDied","Data":"8aa4f390e58846f451cfb1a530266bbf058401bb5967ffc7118acbaf99288533"} Oct 30 15:40:15.584413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.584006 6554 scope.go:122] "RemoveContainer" containerID="8e8ffedaf05e6f8215e3167b17ff77d0a95afbf2d47395709887faaa7c08ac81" Oct 30 15:40:15.584413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.583931 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9" Oct 30 15:40:15.641948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.641920 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9"] Oct 30 15:40:15.655446 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.655420 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-261ad0-6979d7997b-kcxl9"] Oct 30 15:40:15.910904 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.910871 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg"] Oct 30 15:40:15.962972 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.962940 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs"] Oct 30 15:40:15.963164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.963142 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" containerID="cri-o://cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8" gracePeriod=30 Oct 30 15:40:15.976048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:15.976021 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f"] Oct 30 15:40:16.154698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:16.154669 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" path="/var/lib/kubelet/pods/103a7f0d-b730-4d9b-a0a0-b2d29c64dcba/volumes" Oct 30 15:40:19.133240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.133202 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" podUID="7a9ba3eb-543e-436e-8de8-19ccff16926a" containerName="payload-processing" containerID="cri-o://94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02" gracePeriod=25 Oct 30 15:40:19.456039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.456004 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 15:40:19.565633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.565582 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2\" (UniqueName: \"kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2\") pod \"7a9ba3eb-543e-436e-8de8-19ccff16926a\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " Oct 30 15:40:19.565807 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.565690 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume\") pod \"7a9ba3eb-543e-436e-8de8-19ccff16926a\" (UID: \"7a9ba3eb-543e-436e-8de8-19ccff16926a\") " Oct 30 15:40:19.566122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.566093 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume" pod "7a9ba3eb-543e-436e-8de8-19ccff16926a" (UID: "7a9ba3eb-543e-436e-8de8-19ccff16926a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:19.566390 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.566371 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a9ba3eb-543e-436e-8de8-19ccff16926a-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:19.575508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.575475 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2" pod "7a9ba3eb-543e-436e-8de8-19ccff16926a" (UID: "7a9ba3eb-543e-436e-8de8-19ccff16926a"). InnerVolumeSpecName "kube-api-access-rn9t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:19.613511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.613482 6554 generic.go:356] "Generic (PLEG): container finished" podID="7a9ba3eb-543e-436e-8de8-19ccff16926a" containerID="94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02" exitCode=0 Oct 30 15:40:19.613669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.613532 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" Oct 30 15:40:19.613669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.613559 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" event={"ID":"7a9ba3eb-543e-436e-8de8-19ccff16926a","Type":"ContainerDied","Data":"94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02"} Oct 30 15:40:19.613669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.613660 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z" event={"ID":"7a9ba3eb-543e-436e-8de8-19ccff16926a","Type":"ContainerDied","Data":"33ea7e5a1b11efe74232912d57d7cdb10050871c06754a3a3f405fd77774fd1d"} Oct 30 15:40:19.613818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.613699 6554 scope.go:122] "RemoveContainer" containerID="94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02" Oct 30 15:40:19.624536 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.624520 6554 scope.go:122] "RemoveContainer" containerID="94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02" Oct 30 15:40:19.624806 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:19.624787 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02\": container with ID starting with 94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02 not found: ID does not exist" containerID="94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02" Oct 30 15:40:19.624872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.624817 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02"} err="failed to get container status \"94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02\": rpc error: code = NotFound desc = could not find container \"94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02\": container with ID starting with 94b69975285e3774794560cd44f4223acba926b9239085c3b0bb9a01d4583c02 not found: ID does not exist" Oct 30 15:40:19.668202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.668178 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn9t2\" (UniqueName: \"kubernetes.io/projected/7a9ba3eb-543e-436e-8de8-19ccff16926a-kube-api-access-rn9t2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:19.671649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.671625 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z"] Oct 30 15:40:19.676937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:19.676886 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-2c8d89-67655997db-8q86z"] Oct 30 15:40:20.043569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:20.043494 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx"] Oct 30 15:40:20.043737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:20.043701 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" containerID="cri-o://0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189" gracePeriod=30 Oct 30 15:40:20.157397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:20.157360 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a9ba3eb-543e-436e-8de8-19ccff16926a" path="/var/lib/kubelet/pods/7a9ba3eb-543e-436e-8de8-19ccff16926a/volumes" Oct 30 15:40:20.911208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:20.911168 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" podUID="f4b8f84b-141a-451a-801e-5f7bed08dd95" containerName="payload-processing" containerID="cri-o://67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33" gracePeriod=25 Oct 30 15:40:20.977292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:20.977241 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" podUID="f65b66cd-636f-4919-9eaa-d6b8404ef5d4" containerName="payload-pre-processing" containerID="cri-o://1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51" gracePeriod=25 Oct 30 15:40:21.269608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.269577 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 15:40:21.273221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.273200 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:40:21.304987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.304968 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:40:21.394563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394534 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.394736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394575 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.394736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394617 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.394736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394671 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8\" (UniqueName: \"kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8\") pod \"f4b8f84b-141a-451a-801e-5f7bed08dd95\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " Oct 30 15:40:21.394736 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394721 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394739 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume\") pod \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394788 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394862 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume\") pod \"f4b8f84b-141a-451a-801e-5f7bed08dd95\" (UID: \"f4b8f84b-141a-451a-801e-5f7bed08dd95\") " Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394902 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394921 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394948 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.394984 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.394963 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395024 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl\") pod \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\" (UID: \"f65b66cd-636f-4919-9eaa-d6b8404ef5d4\") " Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395060 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395104 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs\") pod \"b85a8334-2fe2-4991-86b4-d4961aefb039\" (UID: \"b85a8334-2fe2-4991-86b4-d4961aefb039\") " Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395224 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395320 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume" pod "f4b8f84b-141a-451a-801e-5f7bed08dd95" (UID: "f4b8f84b-141a-451a-801e-5f7bed08dd95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395484 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume" pod "f65b66cd-636f-4919-9eaa-d6b8404ef5d4" (UID: "f65b66cd-636f-4919-9eaa-d6b8404ef5d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:21.395711 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395700 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395884 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395914 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b8f84b-141a-451a-801e-5f7bed08dd95-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395935 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395951 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395966 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395982 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b85a8334-2fe2-4991-86b4-d4961aefb039-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.396180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.395997 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.397678 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.397656 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "kube-api-access-t8h7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:21.397974 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.397956 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:21.398216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.398197 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8" pod "f4b8f84b-141a-451a-801e-5f7bed08dd95" (UID: "f4b8f84b-141a-451a-801e-5f7bed08dd95"). InnerVolumeSpecName "kube-api-access-nslt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:21.398270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.398226 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl" pod "f65b66cd-636f-4919-9eaa-d6b8404ef5d4" (UID: "f65b66cd-636f-4919-9eaa-d6b8404ef5d4"). InnerVolumeSpecName "kube-api-access-q4qfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:21.398392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.398343 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:21.398995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.398977 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy" pod "b85a8334-2fe2-4991-86b4-d4961aefb039" (UID: "b85a8334-2fe2-4991-86b4-d4961aefb039"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497018 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497046 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497056 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4qfl\" (UniqueName: \"kubernetes.io/projected/f65b66cd-636f-4919-9eaa-d6b8404ef5d4-kube-api-access-q4qfl\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497065 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497075 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b85a8334-2fe2-4991-86b4-d4961aefb039-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497084 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nslt8\" (UniqueName: \"kubernetes.io/projected/f4b8f84b-141a-451a-801e-5f7bed08dd95-kube-api-access-nslt8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.497097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.497094 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8h7s\" (UniqueName: \"kubernetes.io/projected/b85a8334-2fe2-4991-86b4-d4961aefb039-kube-api-access-t8h7s\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:21.632908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.632875 6554 generic.go:356] "Generic (PLEG): container finished" podID="f4b8f84b-141a-451a-801e-5f7bed08dd95" containerID="67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33" exitCode=0 Oct 30 15:40:21.633086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.632919 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" Oct 30 15:40:21.633086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.632956 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" event={"ID":"f4b8f84b-141a-451a-801e-5f7bed08dd95","Type":"ContainerDied","Data":"67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33"} Oct 30 15:40:21.633086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.633011 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg" event={"ID":"f4b8f84b-141a-451a-801e-5f7bed08dd95","Type":"ContainerDied","Data":"05fd1e425c14a0502c3156ec5dc914cd8e81c116bc92da301908dc71a051f797"} Oct 30 15:40:21.633086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.633041 6554 scope.go:122] "RemoveContainer" containerID="67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33" Oct 30 15:40:21.634970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.634944 6554 generic.go:356] "Generic (PLEG): container finished" podID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerID="cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8" exitCode=0 Oct 30 15:40:21.635103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.634972 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" event={"ID":"b85a8334-2fe2-4991-86b4-d4961aefb039","Type":"ContainerDied","Data":"cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8"} Oct 30 15:40:21.635103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.635002 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" Oct 30 15:40:21.635103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.635033 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs" event={"ID":"b85a8334-2fe2-4991-86b4-d4961aefb039","Type":"ContainerDied","Data":"de71fd69f32a808437d4e8b8a920bd03638ba8f930e063c29dff6b865b308f4f"} Oct 30 15:40:21.636576 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.636550 6554 generic.go:356] "Generic (PLEG): container finished" podID="f65b66cd-636f-4919-9eaa-d6b8404ef5d4" containerID="1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51" exitCode=0 Oct 30 15:40:21.636818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.636608 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" Oct 30 15:40:21.636818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.636621 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" event={"ID":"f65b66cd-636f-4919-9eaa-d6b8404ef5d4","Type":"ContainerDied","Data":"1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51"} Oct 30 15:40:21.636818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.636670 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f" event={"ID":"f65b66cd-636f-4919-9eaa-d6b8404ef5d4","Type":"ContainerDied","Data":"45a1a995eaf6dd4262b58e0628883c9a1c5b7b33db2eb4470ace91f6f95f2764"} Oct 30 15:40:21.643909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.643886 6554 scope.go:122] "RemoveContainer" containerID="67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33" Oct 30 15:40:21.644179 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:21.644154 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33\": container with ID starting with 67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33 not found: ID does not exist" containerID="67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33" Oct 30 15:40:21.644265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.644190 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33"} err="failed to get container status \"67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33\": rpc error: code = NotFound desc = could not find container \"67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33\": container with ID starting with 67d9f16cf84c64026d2c7a122e43c1f6231ea84ba2038634e61f7fef0d50ce33 not found: ID does not exist" Oct 30 15:40:21.644265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.644213 6554 scope.go:122] "RemoveContainer" containerID="cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8" Oct 30 15:40:21.654381 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.654362 6554 scope.go:122] "RemoveContainer" containerID="f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c" Oct 30 15:40:21.662672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.662652 6554 scope.go:122] "RemoveContainer" containerID="cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8" Oct 30 15:40:21.662905 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:21.662888 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8\": container with ID starting with cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8 not found: ID does not exist" containerID="cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8" Oct 30 15:40:21.662959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.662911 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8"} err="failed to get container status \"cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8\": rpc error: code = NotFound desc = could not find container \"cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8\": container with ID starting with cd8830199f8ba60967140aa5a035db62f2bb7e7c069497eba4180109fc9a82d8 not found: ID does not exist" Oct 30 15:40:21.662959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.662926 6554 scope.go:122] "RemoveContainer" containerID="f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c" Oct 30 15:40:21.663116 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:21.663101 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c\": container with ID starting with f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c not found: ID does not exist" containerID="f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c" Oct 30 15:40:21.663158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.663120 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c"} err="failed to get container status \"f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c\": rpc error: code = NotFound desc = could not find container \"f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c\": container with ID starting with f63917b332e0573365b381ae96198c7ac8d9807793bdcddf0e0f94605adfc88c not found: ID does not exist" Oct 30 15:40:21.663158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.663134 6554 scope.go:122] "RemoveContainer" containerID="1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51" Oct 30 15:40:21.671399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.671341 6554 scope.go:122] "RemoveContainer" containerID="1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51" Oct 30 15:40:21.671663 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:21.671645 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51\": container with ID starting with 1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51 not found: ID does not exist" containerID="1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51" Oct 30 15:40:21.671738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.671670 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51"} err="failed to get container status \"1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51\": rpc error: code = NotFound desc = could not find container \"1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51\": container with ID starting with 1c0d052e8425f31902b22cc7cbb68095539dbcaf74fdf160f9b361c3eb81fe51 not found: ID does not exist" Oct 30 15:40:21.690089 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.690067 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg"] Oct 30 15:40:21.695983 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.695964 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-01c89d-b9589d44-wt6bg"] Oct 30 15:40:21.735096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.735075 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs"] Oct 30 15:40:21.739024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.739005 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-05d396-openshift-default-7d555f6967-k24xs"] Oct 30 15:40:21.792426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.792372 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f"] Oct 30 15:40:21.796435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:21.796412 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-01c89d-55c9f8fc47-75p8f"] Oct 30 15:40:22.155332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:22.155305 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" path="/var/lib/kubelet/pods/b85a8334-2fe2-4991-86b4-d4961aefb039/volumes" Oct 30 15:40:22.155725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:22.155710 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b8f84b-141a-451a-801e-5f7bed08dd95" path="/var/lib/kubelet/pods/f4b8f84b-141a-451a-801e-5f7bed08dd95/volumes" Oct 30 15:40:22.156069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:22.156056 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f65b66cd-636f-4919-9eaa-d6b8404ef5d4" path="/var/lib/kubelet/pods/f65b66cd-636f-4919-9eaa-d6b8404ef5d4/volumes" Oct 30 15:40:23.503614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.503563 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" containerID="cri-o://3c7c8d4f12ac86127b1bd4085094b93aa32bbfc70abcb3057660b173668e5087" gracePeriod=45 Oct 30 15:40:23.538618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.538579 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" containerID="cri-o://91b55505b5c04a2303a8f8e00fde6e2e7cd65ac5b8b3eec65f8b17c24af315c4" gracePeriod=45 Oct 30 15:40:23.544735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.544709 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" containerID="cri-o://1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4" gracePeriod=45 Oct 30 15:40:23.653112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.653083 6554 generic.go:356] "Generic (PLEG): container finished" podID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerID="3c7c8d4f12ac86127b1bd4085094b93aa32bbfc70abcb3057660b173668e5087" exitCode=0 Oct 30 15:40:23.653230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.653116 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" event={"ID":"f350623a-6add-4f8d-a000-de0cc57ca3af","Type":"ContainerDied","Data":"3c7c8d4f12ac86127b1bd4085094b93aa32bbfc70abcb3057660b173668e5087"} Oct 30 15:40:23.655158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.655138 6554 generic.go:356] "Generic (PLEG): container finished" podID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerID="91b55505b5c04a2303a8f8e00fde6e2e7cd65ac5b8b3eec65f8b17c24af315c4" exitCode=0 Oct 30 15:40:23.655272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.655191 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" event={"ID":"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0","Type":"ContainerDied","Data":"91b55505b5c04a2303a8f8e00fde6e2e7cd65ac5b8b3eec65f8b17c24af315c4"} Oct 30 15:40:23.824710 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.824682 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:40:23.846742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.845169 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:40:23.849686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.849657 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:40:23.924595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924561 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache\") pod \"4617f1de-b33b-4240-b155-972e4e8cf425\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " Oct 30 15:40:23.924776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924603 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir\") pod \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " Oct 30 15:40:23.924776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924654 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm\" (UniqueName: \"kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm\") pod \"4617f1de-b33b-4240-b155-972e4e8cf425\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " Oct 30 15:40:23.924776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924674 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm\") pod \"f350623a-6add-4f8d-a000-de0cc57ca3af\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " Oct 30 15:40:23.924776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924698 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home\") pod \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " Oct 30 15:40:23.924776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924732 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs\" (UniqueName: \"kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs\") pod \"4617f1de-b33b-4240-b155-972e4e8cf425\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " Oct 30 15:40:23.924776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924764 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm\") pod \"4617f1de-b33b-4240-b155-972e4e8cf425\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924790 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs\" (UniqueName: \"kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs\") pod \"f350623a-6add-4f8d-a000-de0cc57ca3af\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924828 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir\") pod \"4617f1de-b33b-4240-b155-972e4e8cf425\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924856 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache\") pod \"f350623a-6add-4f8d-a000-de0cc57ca3af\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924893 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache" pod "4617f1de-b33b-4240-b155-972e4e8cf425" (UID: "4617f1de-b33b-4240-b155-972e4e8cf425"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924960 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd\" (UniqueName: \"kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd\") pod \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924972 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir" pod "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" (UID: "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.924998 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache\") pod \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925029 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm\") pod \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925074 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg\" (UniqueName: \"kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg\") pod \"f350623a-6add-4f8d-a000-de0cc57ca3af\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " Oct 30 15:40:23.925129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925128 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home\") pod \"f350623a-6add-4f8d-a000-de0cc57ca3af\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925167 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs\") pod \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\" (UID: \"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0\") " Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925191 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir" pod "4617f1de-b33b-4240-b155-972e4e8cf425" (UID: "4617f1de-b33b-4240-b155-972e4e8cf425"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925275 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir\") pod \"f350623a-6add-4f8d-a000-de0cc57ca3af\" (UID: \"f350623a-6add-4f8d-a000-de0cc57ca3af\") " Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925328 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home\") pod \"4617f1de-b33b-4240-b155-972e4e8cf425\" (UID: \"4617f1de-b33b-4240-b155-972e4e8cf425\") " Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925404 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache" pod "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" (UID: "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925515 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache" pod "f350623a-6add-4f8d-a000-de0cc57ca3af" (UID: "f350623a-6add-4f8d-a000-de0cc57ca3af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925562 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home" pod "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" (UID: "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.925719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925659 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home" pod "f350623a-6add-4f8d-a000-de0cc57ca3af" (UID: "f350623a-6add-4f8d-a000-de0cc57ca3af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.925742 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home" pod "4617f1de-b33b-4240-b155-972e4e8cf425" (UID: "4617f1de-b33b-4240-b155-972e4e8cf425"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926034 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926054 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926068 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926081 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926094 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926106 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926118 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.926173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926130 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:23.927404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.926265 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir" pod "f350623a-6add-4f8d-a000-de0cc57ca3af" (UID: "f350623a-6add-4f8d-a000-de0cc57ca3af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.929365 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.929327 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd" pod "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" (UID: "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0"). InnerVolumeSpecName "kube-api-access-chvfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:23.929467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.929381 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs" pod "4617f1de-b33b-4240-b155-972e4e8cf425" (UID: "4617f1de-b33b-4240-b155-972e4e8cf425"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:23.929527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.929500 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm" pod "f350623a-6add-4f8d-a000-de0cc57ca3af" (UID: "f350623a-6add-4f8d-a000-de0cc57ca3af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.929606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.929521 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm" pod "4617f1de-b33b-4240-b155-972e4e8cf425" (UID: "4617f1de-b33b-4240-b155-972e4e8cf425"). InnerVolumeSpecName "kube-api-access-bzlpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:23.930205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.930178 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs" pod "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" (UID: "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:23.930547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.930526 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs" pod "f350623a-6add-4f8d-a000-de0cc57ca3af" (UID: "f350623a-6add-4f8d-a000-de0cc57ca3af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:23.930794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.930779 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm" pod "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" (UID: "89ca9f3c-fcaf-4848-b91e-a04f8e9352f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.930928 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.930911 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm" pod "4617f1de-b33b-4240-b155-972e4e8cf425" (UID: "4617f1de-b33b-4240-b155-972e4e8cf425"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:23.931298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:23.931282 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg" pod "f350623a-6add-4f8d-a000-de0cc57ca3af" (UID: "f350623a-6add-4f8d-a000-de0cc57ca3af"). InnerVolumeSpecName "kube-api-access-bj8tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:24.026779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026747 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvfd\" (UniqueName: \"kubernetes.io/projected/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-kube-api-access-chvfd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026778 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026795 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj8tg\" (UniqueName: \"kubernetes.io/projected/f350623a-6add-4f8d-a000-de0cc57ca3af-kube-api-access-bj8tg\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026808 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026821 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026833 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzlpm\" (UniqueName: \"kubernetes.io/projected/4617f1de-b33b-4240-b155-972e4e8cf425-kube-api-access-bzlpm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026846 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f350623a-6add-4f8d-a000-de0cc57ca3af-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026857 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4617f1de-b33b-4240-b155-972e4e8cf425-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026867 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4617f1de-b33b-4240-b155-972e4e8cf425-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.026999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.026878 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f350623a-6add-4f8d-a000-de0cc57ca3af-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:24.662413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.662389 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" Oct 30 15:40:24.662413 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.662402 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl" event={"ID":"89ca9f3c-fcaf-4848-b91e-a04f8e9352f0","Type":"ContainerDied","Data":"436a7729601f6f6ae0d6bd03c1bb5b9e6401ca060447ae68fa5e7f2d29211169"} Oct 30 15:40:24.662917 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.662454 6554 scope.go:122] "RemoveContainer" containerID="91b55505b5c04a2303a8f8e00fde6e2e7cd65ac5b8b3eec65f8b17c24af315c4" Oct 30 15:40:24.663778 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.663756 6554 generic.go:356] "Generic (PLEG): container finished" podID="4617f1de-b33b-4240-b155-972e4e8cf425" containerID="1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4" exitCode=0 Oct 30 15:40:24.663847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.663798 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" Oct 30 15:40:24.663847 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.663830 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" event={"ID":"4617f1de-b33b-4240-b155-972e4e8cf425","Type":"ContainerDied","Data":"1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4"} Oct 30 15:40:24.663963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.663870 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh" event={"ID":"4617f1de-b33b-4240-b155-972e4e8cf425","Type":"ContainerDied","Data":"77f7a995f0302b34202ebaff9d3f4f30db09dcbbaf5a01755a0d043a113c07fa"} Oct 30 15:40:24.665568 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.665545 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" Oct 30 15:40:24.665708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.665570 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" event={"ID":"f350623a-6add-4f8d-a000-de0cc57ca3af","Type":"ContainerDied","Data":"fb39395f74a49508d31629c61892be4cf1a4bb0c3a082894da4a71eb18fe3549"} Oct 30 15:40:24.672517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.672501 6554 scope.go:122] "RemoveContainer" containerID="1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4" Oct 30 15:40:24.680293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.680276 6554 scope.go:122] "RemoveContainer" containerID="1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4" Oct 30 15:40:24.680562 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:24.680542 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4\": container with ID starting with 1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4 not found: ID does not exist" containerID="1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4" Oct 30 15:40:24.680636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.680572 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4"} err="failed to get container status \"1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4\": rpc error: code = NotFound desc = could not find container \"1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4\": container with ID starting with 1aa6b2ec879157f5d63483707bbf90372b4512f352258a16c9357b9d306b43d4 not found: ID does not exist" Oct 30 15:40:24.680636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.680588 6554 scope.go:122] "RemoveContainer" containerID="3c7c8d4f12ac86127b1bd4085094b93aa32bbfc70abcb3057660b173668e5087" Oct 30 15:40:24.718680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.718656 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl"] Oct 30 15:40:24.732052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.732032 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-w6-2c8d89-kserve-69659f7f75-wdkxl"] Oct 30 15:40:24.759325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.759260 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4"] Oct 30 15:40:24.761511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.761491 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4"] Oct 30 15:40:24.797769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.797649 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-2c8d89/distinct-2-w6-2c8d89-kserve-959d85f5b-9ggx4" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 15:40:24.799926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.799907 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh"] Oct 30 15:40:24.804955 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:24.804935 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-2c8d89/simulator-w6-2c8d89-kserve-5f6d56469-9flkh"] Oct 30 15:40:25.311905 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.311851 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:40:25.441282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441255 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441291 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441335 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441407 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441446 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441492 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441635 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441663 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441721 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:25.441918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441737 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token\") pod \"e58e5fda-c858-4072-8631-307319269c5e\" (UID: \"e58e5fda-c858-4072-8631-307319269c5e\") " Oct 30 15:40:25.441918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441761 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:25.441918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.441773 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:25.442149 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.442050 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:25.442274 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.442252 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.442374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.442280 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.442374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.442297 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.442374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.442313 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e58e5fda-c858-4072-8631-307319269c5e-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.442569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.442517 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:25.443707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.443682 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "kube-api-access-bjz9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:25.443844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.443779 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:25.443909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.443855 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:25.443909 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.443859 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo" pod "e58e5fda-c858-4072-8631-307319269c5e" (UID: "e58e5fda-c858-4072-8631-307319269c5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:25.543596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.543576 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjz9c\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-kube-api-access-bjz9c\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.543596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.543597 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.543742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.543607 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e58e5fda-c858-4072-8631-307319269c5e-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.543742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.543617 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e58e5fda-c858-4072-8631-307319269c5e-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.543742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.543625 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e58e5fda-c858-4072-8631-307319269c5e-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:25.674505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.674481 6554 generic.go:356] "Generic (PLEG): container finished" podID="e58e5fda-c858-4072-8631-307319269c5e" containerID="0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189" exitCode=0 Oct 30 15:40:25.674892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.674548 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" event={"ID":"e58e5fda-c858-4072-8631-307319269c5e","Type":"ContainerDied","Data":"0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189"} Oct 30 15:40:25.674892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.674553 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" Oct 30 15:40:25.674892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.674590 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx" event={"ID":"e58e5fda-c858-4072-8631-307319269c5e","Type":"ContainerDied","Data":"101982bb2db41151cad4dad928be65413fb6bb37b0a34ec71da38d4d9f5637f4"} Oct 30 15:40:25.674892 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.674615 6554 scope.go:122] "RemoveContainer" containerID="0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189" Oct 30 15:40:25.685017 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.684999 6554 scope.go:122] "RemoveContainer" containerID="11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c" Oct 30 15:40:25.693870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.693855 6554 scope.go:122] "RemoveContainer" containerID="0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189" Oct 30 15:40:25.694123 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:25.694100 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189\": container with ID starting with 0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189 not found: ID does not exist" containerID="0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189" Oct 30 15:40:25.694168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.694129 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189"} err="failed to get container status \"0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189\": rpc error: code = NotFound desc = could not find container \"0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189\": container with ID starting with 0ef0f8c488be1c1c0a6072dee8f7c4895d593ea5b05b17bffe335a06cf558189 not found: ID does not exist" Oct 30 15:40:25.694168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.694145 6554 scope.go:122] "RemoveContainer" containerID="11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c" Oct 30 15:40:25.694401 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:25.694382 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c\": container with ID starting with 11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c not found: ID does not exist" containerID="11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c" Oct 30 15:40:25.694457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.694410 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c"} err="failed to get container status \"11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c\": rpc error: code = NotFound desc = could not find container \"11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c\": container with ID starting with 11ee5ab0be5d9e1c18622cb909210b42c7c3385024149319859f6bce5d59df3c not found: ID does not exist" Oct 30 15:40:25.732232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.732212 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx"] Oct 30 15:40:25.745677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:25.745656 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e30f62-openshift-default-64d6955b85-8bzpx"] Oct 30 15:40:26.154713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:26.154683 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" path="/var/lib/kubelet/pods/4617f1de-b33b-4240-b155-972e4e8cf425/volumes" Oct 30 15:40:26.155093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:26.155079 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" path="/var/lib/kubelet/pods/89ca9f3c-fcaf-4848-b91e-a04f8e9352f0/volumes" Oct 30 15:40:26.155510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:26.155496 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e58e5fda-c858-4072-8631-307319269c5e" path="/var/lib/kubelet/pods/e58e5fda-c858-4072-8631-307319269c5e/volumes" Oct 30 15:40:26.156179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:26.156165 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" path="/var/lib/kubelet/pods/f350623a-6add-4f8d-a000-de0cc57ca3af/volumes" Oct 30 15:40:31.014160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:31.014125 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm"] Oct 30 15:40:31.048395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:31.048367 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8"] Oct 30 15:40:33.042668 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:33.042638 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf"] Oct 30 15:40:36.014778 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:36.014717 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerName="payload-processing" containerID="cri-o://96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b" gracePeriod=25 Oct 30 15:40:36.049174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:36.049141 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerName="payload-pre-processing" containerID="cri-o://a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730" gracePeriod=25 Oct 30 15:40:38.264072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:38.264038 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:38.264515 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:38.264046 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:38.670278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:38.670250 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f"] Oct 30 15:40:38.703088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:38.703059 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d"] Oct 30 15:40:39.201810 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:39.201771 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:40:43.671035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:43.670988 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerName="payload-processing" containerID="cri-o://b6aaef3402c9c24f6cbd9e78e8948206e5c03ea052bec03e691b5dc80581448b" gracePeriod=25 Oct 30 15:40:43.703492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:43.703453 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" podUID="629da890-dde2-471f-a304-acf45a058a8d" containerName="payload-pre-processing" containerID="cri-o://644a5c9b17c6b87d6f4886b246ea7d72a2ed65ee4de9b468350c022d17db7680" gracePeriod=25 Oct 30 15:40:46.218129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:46.218090 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:46.218129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:46.218121 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" podUID="629da890-dde2-471f-a304-acf45a058a8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:48.063333 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.063296 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" containerID="cri-o://be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9" gracePeriod=45 Oct 30 15:40:48.264610 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.264568 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:48.264783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.264620 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:48.322774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.322720 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:40:48.406691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.406574 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs\") pod \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " Oct 30 15:40:48.406691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.406639 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache\") pod \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " Oct 30 15:40:48.407236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.406992 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm\") pod \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " Oct 30 15:40:48.407236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407055 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache" pod "cb90950f-4c23-4af9-b33a-4408a5d1ec68" (UID: "cb90950f-4c23-4af9-b33a-4408a5d1ec68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.407236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407100 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir\") pod \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " Oct 30 15:40:48.407236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407173 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t\" (UniqueName: \"kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t\") pod \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " Oct 30 15:40:48.407860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407261 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home\") pod \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\" (UID: \"cb90950f-4c23-4af9-b33a-4408a5d1ec68\") " Oct 30 15:40:48.407860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407449 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir" pod "cb90950f-4c23-4af9-b33a-4408a5d1ec68" (UID: "cb90950f-4c23-4af9-b33a-4408a5d1ec68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.407860 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407575 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home" pod "cb90950f-4c23-4af9-b33a-4408a5d1ec68" (UID: "cb90950f-4c23-4af9-b33a-4408a5d1ec68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.408059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407854 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.408059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407878 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.408059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.407893 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.409772 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.409745 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm" pod "cb90950f-4c23-4af9-b33a-4408a5d1ec68" (UID: "cb90950f-4c23-4af9-b33a-4408a5d1ec68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:48.409881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.409848 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs" pod "cb90950f-4c23-4af9-b33a-4408a5d1ec68" (UID: "cb90950f-4c23-4af9-b33a-4408a5d1ec68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:40:48.410169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.410149 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t" pod "cb90950f-4c23-4af9-b33a-4408a5d1ec68" (UID: "cb90950f-4c23-4af9-b33a-4408a5d1ec68"). InnerVolumeSpecName "kube-api-access-6t75t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:48.508739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.508713 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t75t\" (UniqueName: \"kubernetes.io/projected/cb90950f-4c23-4af9-b33a-4408a5d1ec68-kube-api-access-6t75t\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.508739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.508739 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb90950f-4c23-4af9-b33a-4408a5d1ec68-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.508956 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.508750 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb90950f-4c23-4af9-b33a-4408a5d1ec68-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:48.819825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.819791 6554 generic.go:356] "Generic (PLEG): container finished" podID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerID="be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9" exitCode=0 Oct 30 15:40:48.820011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.819831 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" Oct 30 15:40:48.820011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.819867 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" event={"ID":"cb90950f-4c23-4af9-b33a-4408a5d1ec68","Type":"ContainerDied","Data":"be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9"} Oct 30 15:40:48.820011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.819910 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf" event={"ID":"cb90950f-4c23-4af9-b33a-4408a5d1ec68","Type":"ContainerDied","Data":"aea9ee83e9ca64833c9016fa55ddf79c615c13201b4364b4819bdaac090a3d2e"} Oct 30 15:40:48.820011 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.819935 6554 scope.go:122] "RemoveContainer" containerID="be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9" Oct 30 15:40:48.829650 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.829630 6554 scope.go:122] "RemoveContainer" containerID="be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9" Oct 30 15:40:48.829921 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:48.829900 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9\": container with ID starting with be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9 not found: ID does not exist" containerID="be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9" Oct 30 15:40:48.829996 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.829929 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9"} err="failed to get container status \"be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9\": rpc error: code = NotFound desc = could not find container \"be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9\": container with ID starting with be941646b656261d4772e94bba5af84461a093eabb2bd18a6486634c58e4e9a9 not found: ID does not exist" Oct 30 15:40:48.876951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.876928 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf"] Oct 30 15:40:48.890332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:48.890310 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-357948/distinct-w3-357948-kserve-86c4db5f86-2lshf"] Oct 30 15:40:50.154071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:50.154035 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" path="/var/lib/kubelet/pods/cb90950f-4c23-4af9-b33a-4408a5d1ec68/volumes" Oct 30 15:40:50.716918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:50.716877 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf"] Oct 30 15:40:50.717155 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:50.717130 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" containerID="cri-o://d6b3a26c15bc77fdd98108f45b6fc97892f598d9251dece8099422e56fcc2160" gracePeriod=30 Oct 30 15:40:55.866432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:55.866399 6554 generic.go:356] "Generic (PLEG): container finished" podID="146b4be7-602b-4930-9bed-d94446ea35b4" containerID="d6b3a26c15bc77fdd98108f45b6fc97892f598d9251dece8099422e56fcc2160" exitCode=0 Oct 30 15:40:55.866770 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:55.866474 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" event={"ID":"146b4be7-602b-4930-9bed-d94446ea35b4","Type":"ContainerDied","Data":"d6b3a26c15bc77fdd98108f45b6fc97892f598d9251dece8099422e56fcc2160"} Oct 30 15:40:55.980440 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:55.980386 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:40:56.106938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.106912 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.106965 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.106993 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107048 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107096 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107117 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107177 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107210 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107219 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.107452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107300 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert\") pod \"146b4be7-602b-4930-9bed-d94446ea35b4\" (UID: \"146b4be7-602b-4930-9bed-d94446ea35b4\") " Oct 30 15:40:56.107452 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107340 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.107715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107491 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.107715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107505 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.107844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107823 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:56.107901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107866 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.107901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107888 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.107980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107902 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.107980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.107918 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.109481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.109458 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "kube-api-access-sg2zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.109481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.109471 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.109642 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.109627 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:40:56.109688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.109645 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo" pod "146b4be7-602b-4930-9bed-d94446ea35b4" (UID: "146b4be7-602b-4930-9bed-d94446ea35b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:40:56.209171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.209137 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b4be7-602b-4930-9bed-d94446ea35b4-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.209171 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.209168 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg2zd\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-kube-api-access-sg2zd\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.209391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.209181 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b4be7-602b-4930-9bed-d94446ea35b4-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.209391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.209189 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b4be7-602b-4930-9bed-d94446ea35b4-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.209391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.209197 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b4be7-602b-4930-9bed-d94446ea35b4-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.222794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.222756 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" podUID="629da890-dde2-471f-a304-acf45a058a8d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:56.225676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.225645 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:40:56.295799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.295771 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:40:56.351958 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.351934 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:40:56.411679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.411654 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts\" (UniqueName: \"kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts\") pod \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " Oct 30 15:40:56.411849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.411831 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume\") pod \"45d8c605-42eb-4c52-850c-c7e6863c6753\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " Oct 30 15:40:56.411899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.411871 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj\" (UniqueName: \"kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj\") pod \"45d8c605-42eb-4c52-850c-c7e6863c6753\" (UID: \"45d8c605-42eb-4c52-850c-c7e6863c6753\") " Oct 30 15:40:56.411944 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.411907 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume\") pod \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\" (UID: \"b3d8f8af-2bbc-47b5-ae69-a05f33866190\") " Oct 30 15:40:56.412317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.412275 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume" pod "45d8c605-42eb-4c52-850c-c7e6863c6753" (UID: "45d8c605-42eb-4c52-850c-c7e6863c6753"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:56.412317 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.412294 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume" pod "b3d8f8af-2bbc-47b5-ae69-a05f33866190" (UID: "b3d8f8af-2bbc-47b5-ae69-a05f33866190"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:40:56.414331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.414307 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj" pod "45d8c605-42eb-4c52-850c-c7e6863c6753" (UID: "45d8c605-42eb-4c52-850c-c7e6863c6753"). InnerVolumeSpecName "kube-api-access-7rxqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.414471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.414325 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts" pod "b3d8f8af-2bbc-47b5-ae69-a05f33866190" (UID: "b3d8f8af-2bbc-47b5-ae69-a05f33866190"). InnerVolumeSpecName "kube-api-access-gpjts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:40:56.513641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.513614 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45d8c605-42eb-4c52-850c-c7e6863c6753-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.513641 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.513639 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rxqj\" (UniqueName: \"kubernetes.io/projected/45d8c605-42eb-4c52-850c-c7e6863c6753-kube-api-access-7rxqj\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.513795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.513653 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3d8f8af-2bbc-47b5-ae69-a05f33866190-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.513795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.513667 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpjts\" (UniqueName: \"kubernetes.io/projected/b3d8f8af-2bbc-47b5-ae69-a05f33866190-kube-api-access-gpjts\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:40:56.873614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.873541 6554 generic.go:356] "Generic (PLEG): container finished" podID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerID="96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b" exitCode=0 Oct 30 15:40:56.873614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.873584 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" Oct 30 15:40:56.874096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.873624 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" event={"ID":"b3d8f8af-2bbc-47b5-ae69-a05f33866190","Type":"ContainerDied","Data":"96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b"} Oct 30 15:40:56.874096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.873687 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm" event={"ID":"b3d8f8af-2bbc-47b5-ae69-a05f33866190","Type":"ContainerDied","Data":"0cc2e7ab71db95a70657191a010b0c22d29913e4b5ac26a80ffed70087a21feb"} Oct 30 15:40:56.874096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.873719 6554 scope.go:122] "RemoveContainer" containerID="96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b" Oct 30 15:40:56.875303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.875288 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" Oct 30 15:40:56.875429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.875337 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf" event={"ID":"146b4be7-602b-4930-9bed-d94446ea35b4","Type":"ContainerDied","Data":"aa778791df8fa45169a72dfffd9e99cc37b5c1f95620bcfaf4e9fc9098d14da5"} Oct 30 15:40:56.876919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.876893 6554 generic.go:356] "Generic (PLEG): container finished" podID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerID="a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730" exitCode=0 Oct 30 15:40:56.877023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.876930 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" Oct 30 15:40:56.877023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.876972 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" event={"ID":"45d8c605-42eb-4c52-850c-c7e6863c6753","Type":"ContainerDied","Data":"a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730"} Oct 30 15:40:56.877119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.877022 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8" event={"ID":"45d8c605-42eb-4c52-850c-c7e6863c6753","Type":"ContainerDied","Data":"84eafa53990003e36a6427b8afc3c20389ad4d6de5ec035b5969e5e26dabb216"} Oct 30 15:40:56.884861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.884844 6554 scope.go:122] "RemoveContainer" containerID="96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b" Oct 30 15:40:56.885143 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:56.885119 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b\": container with ID starting with 96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b not found: ID does not exist" containerID="96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b" Oct 30 15:40:56.885189 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.885149 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b"} err="failed to get container status \"96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b\": rpc error: code = NotFound desc = could not find container \"96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b\": container with ID starting with 96036d5538d3d904fe7e28583123ef5b3a23d4bea661d9f658ba3344f064e82b not found: ID does not exist" Oct 30 15:40:56.885189 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.885165 6554 scope.go:122] "RemoveContainer" containerID="d6b3a26c15bc77fdd98108f45b6fc97892f598d9251dece8099422e56fcc2160" Oct 30 15:40:56.894177 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.894163 6554 scope.go:122] "RemoveContainer" containerID="a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730" Oct 30 15:40:56.902260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.902242 6554 scope.go:122] "RemoveContainer" containerID="a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730" Oct 30 15:40:56.902533 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:40:56.902514 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730\": container with ID starting with a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730 not found: ID does not exist" containerID="a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730" Oct 30 15:40:56.902605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.902536 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730"} err="failed to get container status \"a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730\": rpc error: code = NotFound desc = could not find container \"a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730\": container with ID starting with a3e0f9d8ee343c93b09e49edd82b781f46ff7f21114224aaa6ea4d555d96f730 not found: ID does not exist" Oct 30 15:40:56.930104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.930080 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf"] Oct 30 15:40:56.943723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.943702 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-bcd8d6-openshift-default-848796477c-vtwgf"] Oct 30 15:40:56.970654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.970631 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8"] Oct 30 15:40:56.974462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:56.974441 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-3f20f9-d8fd995ff-kl6c8"] Oct 30 15:40:57.010858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:57.010805 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm"] Oct 30 15:40:57.016705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:57.016686 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-3f20f9-64d57dcc9b-tl5jm"] Oct 30 15:40:58.153334 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:58.153301 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" path="/var/lib/kubelet/pods/146b4be7-602b-4930-9bed-d94446ea35b4/volumes" Oct 30 15:40:58.153744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:58.153713 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" path="/var/lib/kubelet/pods/45d8c605-42eb-4c52-850c-c7e6863c6753/volumes" Oct 30 15:40:58.154062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:40:58.154047 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" path="/var/lib/kubelet/pods/b3d8f8af-2bbc-47b5-ae69-a05f33866190/volumes" Oct 30 15:41:01.812557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:01.812517 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7"] Oct 30 15:41:01.813068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:01.812780 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" podUID="f5ceb578-984d-43cd-8797-50a286dea383" containerName="maas-api" containerID="cri-o://b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2" gracePeriod=30 Oct 30 15:41:01.843036 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:01.843010 6554 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ceb578_984d_43cd_8797_50a286dea383.slice/crio-conmon-b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2.scope\": RecentStats: unable to find data in memory cache]" Oct 30 15:41:02.095082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.095054 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:41:02.172006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.171971 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr\" (UniqueName: \"kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr\") pod \"f5ceb578-984d-43cd-8797-50a286dea383\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " Oct 30 15:41:02.172190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.172112 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls\") pod \"f5ceb578-984d-43cd-8797-50a286dea383\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " Oct 30 15:41:02.172263 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.172246 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca\") pod \"f5ceb578-984d-43cd-8797-50a286dea383\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " Oct 30 15:41:02.172327 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.172280 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs\") pod \"f5ceb578-984d-43cd-8797-50a286dea383\" (UID: \"f5ceb578-984d-43cd-8797-50a286dea383\") " Oct 30 15:41:02.172619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.172592 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca" pod "f5ceb578-984d-43cd-8797-50a286dea383" (UID: "f5ceb578-984d-43cd-8797-50a286dea383"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:02.172890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.172863 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5ceb578-984d-43cd-8797-50a286dea383-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.174449 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.174425 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs" pod "f5ceb578-984d-43cd-8797-50a286dea383" (UID: "f5ceb578-984d-43cd-8797-50a286dea383"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:02.174530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.174443 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls" pod "f5ceb578-984d-43cd-8797-50a286dea383" (UID: "f5ceb578-984d-43cd-8797-50a286dea383"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:02.174616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.174600 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr" pod "f5ceb578-984d-43cd-8797-50a286dea383" (UID: "f5ceb578-984d-43cd-8797-50a286dea383"). InnerVolumeSpecName "kube-api-access-zwclr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:02.273729 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.273706 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.273811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.273732 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5ceb578-984d-43cd-8797-50a286dea383-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.273811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.273746 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwclr\" (UniqueName: \"kubernetes.io/projected/f5ceb578-984d-43cd-8797-50a286dea383-kube-api-access-zwclr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:02.924999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.924966 6554 generic.go:356] "Generic (PLEG): container finished" podID="f5ceb578-984d-43cd-8797-50a286dea383" containerID="b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2" exitCode=0 Oct 30 15:41:02.925430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.925016 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" Oct 30 15:41:02.925430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.925051 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" event={"ID":"f5ceb578-984d-43cd-8797-50a286dea383","Type":"ContainerDied","Data":"b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2"} Oct 30 15:41:02.925430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.925111 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7" event={"ID":"f5ceb578-984d-43cd-8797-50a286dea383","Type":"ContainerDied","Data":"affde0ab85fd28ec2daf60169a6e793349ba5bd080330b48b07d0cd070ccc116"} Oct 30 15:41:02.925430 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.925125 6554 scope.go:122] "RemoveContainer" containerID="b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2" Oct 30 15:41:02.935280 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.935263 6554 scope.go:122] "RemoveContainer" containerID="b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2" Oct 30 15:41:02.935547 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:02.935527 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2\": container with ID starting with b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2 not found: ID does not exist" containerID="b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2" Oct 30 15:41:02.935646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.935553 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2"} err="failed to get container status \"b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2\": rpc error: code = NotFound desc = could not find container \"b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2\": container with ID starting with b18cbeaffa563c3e9ebd6bd1dbbe8e8c641a4626ea3c74c6532014f24d92f5e2 not found: ID does not exist" Oct 30 15:41:02.984747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.984724 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7"] Oct 30 15:41:02.998475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:02.998456 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-7024f9-9859dfff4-59ct7"] Oct 30 15:41:03.937358 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:03.937307 6554 generic.go:356] "Generic (PLEG): container finished" podID="629da890-dde2-471f-a304-acf45a058a8d" containerID="644a5c9b17c6b87d6f4886b246ea7d72a2ed65ee4de9b468350c022d17db7680" exitCode=0 Oct 30 15:41:03.937735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:03.937402 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" event={"ID":"629da890-dde2-471f-a304-acf45a058a8d","Type":"ContainerDied","Data":"644a5c9b17c6b87d6f4886b246ea7d72a2ed65ee4de9b468350c022d17db7680"} Oct 30 15:41:03.939064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:03.939042 6554 generic.go:356] "Generic (PLEG): container finished" podID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerID="b6aaef3402c9c24f6cbd9e78e8948206e5c03ea052bec03e691b5dc80581448b" exitCode=0 Oct 30 15:41:03.939202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:03.939111 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" event={"ID":"f4fd997d-2b65-49b4-a644-f1438a37c184","Type":"ContainerDied","Data":"b6aaef3402c9c24f6cbd9e78e8948206e5c03ea052bec03e691b5dc80581448b"} Oct 30 15:41:03.958565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:03.958541 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 15:41:03.987823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:03.987805 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 15:41:04.094093 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.094064 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk\" (UniqueName: \"kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk\") pod \"f4fd997d-2b65-49b4-a644-f1438a37c184\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " Oct 30 15:41:04.094261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.094100 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf\" (UniqueName: \"kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf\") pod \"629da890-dde2-471f-a304-acf45a058a8d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " Oct 30 15:41:04.094261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.094123 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume\") pod \"f4fd997d-2b65-49b4-a644-f1438a37c184\" (UID: \"f4fd997d-2b65-49b4-a644-f1438a37c184\") " Oct 30 15:41:04.094261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.094217 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume\") pod \"629da890-dde2-471f-a304-acf45a058a8d\" (UID: \"629da890-dde2-471f-a304-acf45a058a8d\") " Oct 30 15:41:04.094577 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.094550 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume" pod "f4fd997d-2b65-49b4-a644-f1438a37c184" (UID: "f4fd997d-2b65-49b4-a644-f1438a37c184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:04.094662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.094606 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume" pod "629da890-dde2-471f-a304-acf45a058a8d" (UID: "629da890-dde2-471f-a304-acf45a058a8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:04.096261 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.096241 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf" pod "629da890-dde2-471f-a304-acf45a058a8d" (UID: "629da890-dde2-471f-a304-acf45a058a8d"). InnerVolumeSpecName "kube-api-access-5xstf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:04.096372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.096298 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk" pod "f4fd997d-2b65-49b4-a644-f1438a37c184" (UID: "f4fd997d-2b65-49b4-a644-f1438a37c184"). InnerVolumeSpecName "kube-api-access-8hrxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:04.153287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.153262 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ceb578-984d-43cd-8797-50a286dea383" path="/var/lib/kubelet/pods/f5ceb578-984d-43cd-8797-50a286dea383/volumes" Oct 30 15:41:04.195783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.195728 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hrxk\" (UniqueName: \"kubernetes.io/projected/f4fd997d-2b65-49b4-a644-f1438a37c184-kube-api-access-8hrxk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.195783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.195749 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xstf\" (UniqueName: \"kubernetes.io/projected/629da890-dde2-471f-a304-acf45a058a8d-kube-api-access-5xstf\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.195783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.195759 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4fd997d-2b65-49b4-a644-f1438a37c184-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.195783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.195768 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629da890-dde2-471f-a304-acf45a058a8d-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:04.945927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.945841 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" Oct 30 15:41:04.945927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.945876 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f" event={"ID":"f4fd997d-2b65-49b4-a644-f1438a37c184","Type":"ContainerDied","Data":"e95b23e04b8a9ab18d1f928f7ee29929c3ad43efb6a3531fc679bcfc0ab278aa"} Oct 30 15:41:04.946496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.945938 6554 scope.go:122] "RemoveContainer" containerID="b6aaef3402c9c24f6cbd9e78e8948206e5c03ea052bec03e691b5dc80581448b" Oct 30 15:41:04.947426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.947409 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" Oct 30 15:41:04.947515 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.947488 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d" event={"ID":"629da890-dde2-471f-a304-acf45a058a8d","Type":"ContainerDied","Data":"89fe5b78f51087497b7d32057c8f7261f012364fcc9e456363e9c6492d0308ac"} Oct 30 15:41:04.955299 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:04.955282 6554 scope.go:122] "RemoveContainer" containerID="644a5c9b17c6b87d6f4886b246ea7d72a2ed65ee4de9b468350c022d17db7680" Oct 30 15:41:05.001487 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:05.001457 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d"] Oct 30 15:41:05.014674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:05.014651 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-357948-f8cbc7f95-gj24d"] Oct 30 15:41:05.042068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:05.042046 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f"] Oct 30 15:41:05.048084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:05.048065 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-357948-9ccf74696-nw48f"] Oct 30 15:41:06.153473 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:06.153441 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629da890-dde2-471f-a304-acf45a058a8d" path="/var/lib/kubelet/pods/629da890-dde2-471f-a304-acf45a058a8d/volumes" Oct 30 15:41:06.153871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:06.153827 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" path="/var/lib/kubelet/pods/f4fd997d-2b65-49b4-a644-f1438a37c184/volumes" Oct 30 15:41:09.202026 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:09.201991 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:41:14.700371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:14.696496 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt"] Oct 30 15:41:14.700371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:14.696786 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" podUID="236fbc06-3de3-4f3b-9da8-9de010521aa2" containerName="maas-api" containerID="cri-o://7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798" gracePeriod=30 Oct 30 15:41:14.987170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:14.987145 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:41:15.014420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.014390 6554 generic.go:356] "Generic (PLEG): container finished" podID="236fbc06-3de3-4f3b-9da8-9de010521aa2" containerID="7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798" exitCode=0 Oct 30 15:41:15.014541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.014466 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" Oct 30 15:41:15.014541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.014488 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" event={"ID":"236fbc06-3de3-4f3b-9da8-9de010521aa2","Type":"ContainerDied","Data":"7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798"} Oct 30 15:41:15.014541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.014533 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt" event={"ID":"236fbc06-3de3-4f3b-9da8-9de010521aa2","Type":"ContainerDied","Data":"66043a7d6659f614f696a5d87d3294d1180388fe28982e68c9d1f37213b00bd5"} Oct 30 15:41:15.014663 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.014554 6554 scope.go:122] "RemoveContainer" containerID="7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798" Oct 30 15:41:15.024250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.024233 6554 scope.go:122] "RemoveContainer" containerID="7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798" Oct 30 15:41:15.024553 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:15.024530 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798\": container with ID starting with 7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798 not found: ID does not exist" containerID="7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798" Oct 30 15:41:15.024654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.024557 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798"} err="failed to get container status \"7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798\": rpc error: code = NotFound desc = could not find container \"7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798\": container with ID starting with 7b1bd61c31ed4fb7b4918c48b17daf5fe8a0affc67704ecfdfe898d48ddad798 not found: ID does not exist" Oct 30 15:41:15.101713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.101682 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs\") pod \"236fbc06-3de3-4f3b-9da8-9de010521aa2\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " Oct 30 15:41:15.101885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.101718 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\") pod \"236fbc06-3de3-4f3b-9da8-9de010521aa2\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " Oct 30 15:41:15.101885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.101757 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca\") pod \"236fbc06-3de3-4f3b-9da8-9de010521aa2\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " Oct 30 15:41:15.101885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.101774 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n\" (UniqueName: \"kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n\") pod \"236fbc06-3de3-4f3b-9da8-9de010521aa2\" (UID: \"236fbc06-3de3-4f3b-9da8-9de010521aa2\") " Oct 30 15:41:15.102203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.102176 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca" pod "236fbc06-3de3-4f3b-9da8-9de010521aa2" (UID: "236fbc06-3de3-4f3b-9da8-9de010521aa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:15.103916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.103886 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls" pod "236fbc06-3de3-4f3b-9da8-9de010521aa2" (UID: "236fbc06-3de3-4f3b-9da8-9de010521aa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:15.103916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.103909 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n" pod "236fbc06-3de3-4f3b-9da8-9de010521aa2" (UID: "236fbc06-3de3-4f3b-9da8-9de010521aa2"). InnerVolumeSpecName "kube-api-access-b8t2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:15.104054 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.103971 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs" pod "236fbc06-3de3-4f3b-9da8-9de010521aa2" (UID: "236fbc06-3de3-4f3b-9da8-9de010521aa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:15.212387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.203657 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:15.212387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.203691 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/236fbc06-3de3-4f3b-9da8-9de010521aa2-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:15.212387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.203704 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/236fbc06-3de3-4f3b-9da8-9de010521aa2-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:15.212387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.203719 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8t2n\" (UniqueName: \"kubernetes.io/projected/236fbc06-3de3-4f3b-9da8-9de010521aa2-kube-api-access-b8t2n\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:15.390151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.390106 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt"] Oct 30 15:41:15.414656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:15.414626 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-01c89d-6679777b7f-pcnlt"] Oct 30 15:41:16.154718 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:16.154682 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="236fbc06-3de3-4f3b-9da8-9de010521aa2" path="/var/lib/kubelet/pods/236fbc06-3de3-4f3b-9da8-9de010521aa2/volumes" Oct 30 15:41:23.162598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:23.162566 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn"] Oct 30 15:41:23.201583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:23.201558 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv"] Oct 30 15:41:23.340271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:23.340234 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9"] Oct 30 15:41:23.340534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:23.340503 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" containerID="cri-o://aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f" gracePeriod=30 Oct 30 15:41:23.980541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:23.980504 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr"] Oct 30 15:41:23.980814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:23.980762 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" containerID="cri-o://698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30" gracePeriod=30 Oct 30 15:41:24.776174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776140 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g"] Oct 30 15:41:24.776883 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776862 6554 cpu_manager.go:414] "containerMap: removing container" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" Oct 30 15:41:24.776941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776887 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" Oct 30 15:41:24.776941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776916 6554 cpu_manager.go:414] "containerMap: removing container" podUID="73043333-745a-4a8a-a431-6f4b493d1600" containerName="payload-pre-processing" Oct 30 15:41:24.776941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776924 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73043333-745a-4a8a-a431-6f4b493d1600" containerName="payload-pre-processing" Oct 30 15:41:24.776941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776935 6554 cpu_manager.go:414] "containerMap: removing container" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776942 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776955 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerName="payload-processing" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776964 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerName="payload-processing" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776976 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776983 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.776997 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerName="payload-processing" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777005 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerName="payload-processing" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777016 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f65b66cd-636f-4919-9eaa-d6b8404ef5d4" containerName="payload-pre-processing" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777024 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f65b66cd-636f-4919-9eaa-d6b8404ef5d4" containerName="payload-pre-processing" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777038 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777046 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777066 6554 cpu_manager.go:414] "containerMap: removing container" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777073 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" Oct 30 15:41:24.777083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777083 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777090 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777106 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777113 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777126 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f5ceb578-984d-43cd-8797-50a286dea383" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777133 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ceb578-984d-43cd-8797-50a286dea383" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777149 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7a9ba3eb-543e-436e-8de8-19ccff16926a" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777157 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a9ba3eb-543e-436e-8de8-19ccff16926a" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777171 6554 cpu_manager.go:414] "containerMap: removing container" podUID="629da890-dde2-471f-a304-acf45a058a8d" containerName="payload-pre-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777177 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629da890-dde2-471f-a304-acf45a058a8d" containerName="payload-pre-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777192 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777199 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777215 6554 cpu_manager.go:414] "containerMap: removing container" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerName="payload-pre-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777222 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerName="payload-pre-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777233 6554 cpu_manager.go:414] "containerMap: removing container" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777240 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777255 6554 cpu_manager.go:414] "containerMap: removing container" podUID="103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777262 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777285 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b5ea4576-a5ed-494c-a471-eea268e6548a" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777292 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5ea4576-a5ed-494c-a471-eea268e6548a" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777304 6554 cpu_manager.go:414] "containerMap: removing container" podUID="236fbc06-3de3-4f3b-9da8-9de010521aa2" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777311 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="236fbc06-3de3-4f3b-9da8-9de010521aa2" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777322 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777329 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777340 6554 cpu_manager.go:414] "containerMap: removing container" podUID="4afb49d4-4ab0-45c2-8fdc-190326fbcb39" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777373 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4afb49d4-4ab0-45c2-8fdc-190326fbcb39" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777385 6554 cpu_manager.go:414] "containerMap: removing container" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777392 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777401 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f4b8f84b-141a-451a-801e-5f7bed08dd95" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777411 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b8f84b-141a-451a-801e-5f7bed08dd95" containerName="payload-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777527 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777542 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d8c605-42eb-4c52-850c-c7e6863c6753" containerName="payload-pre-processing" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777554 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="89ca9f3c-fcaf-4848-b91e-a04f8e9352f0" containerName="main" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777565 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="103a7f0d-b730-4d9b-a0a0-b2d29c64dcba" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777578 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5ea4576-a5ed-494c-a471-eea268e6548a" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777598 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="236fbc06-3de3-4f3b-9da8-9de010521aa2" containerName="maas-api" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777607 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85a8334-2fe2-4991-86b4-d4961aefb039" containerName="istio-proxy" Oct 30 15:41:24.777602 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777615 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="4afb49d4-4ab0-45c2-8fdc-190326fbcb39" containerName="payload-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777623 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfd8f9b8-62f3-4055-836f-6808267ab03d" containerName="payload-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777631 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b8f84b-141a-451a-801e-5f7bed08dd95" containerName="payload-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777640 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="4617f1de-b33b-4240-b155-972e4e8cf425" containerName="main" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777652 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="73043333-745a-4a8a-a431-6f4b493d1600" containerName="payload-pre-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777659 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="146b4be7-602b-4930-9bed-d94446ea35b4" containerName="istio-proxy" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777669 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fd997d-2b65-49b4-a644-f1438a37c184" containerName="payload-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777676 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777684 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3d8f8af-2bbc-47b5-ae69-a05f33866190" containerName="payload-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777692 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f65b66cd-636f-4919-9eaa-d6b8404ef5d4" containerName="payload-pre-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777702 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f350623a-6add-4f8d-a000-de0cc57ca3af" containerName="main" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777712 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb90950f-4c23-4af9-b33a-4408a5d1ec68" containerName="main" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777720 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e0098b7-a192-4820-9aad-026cb459eb4a" containerName="istio-proxy" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777730 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ceb578-984d-43cd-8797-50a286dea383" containerName="maas-api" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777739 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a9ba3eb-543e-436e-8de8-19ccff16926a" containerName="payload-processing" Oct 30 15:41:24.778776 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.777748 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="629da890-dde2-471f-a304-acf45a058a8d" containerName="payload-pre-processing" Oct 30 15:41:24.783814 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.783789 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.804453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.804426 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g"] Oct 30 15:41:24.813744 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.813719 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:41:24.814016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.813997 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:41:24.896314 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.896291 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.896441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.896345 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzs8c\" (UniqueName: \"kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.896545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.896526 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.997323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.997294 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzs8c\" (UniqueName: \"kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.997439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.997410 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.997484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.997471 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.998023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.997996 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:24.999792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:24.999768 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:25.022453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:25.022432 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzs8c\" (UniqueName: \"kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c\") pod \"maas-api-revoke-sub-eb5cb392-g8h6g\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:25.098554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:25.098507 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:25.402749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:25.402722 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g"] Oct 30 15:41:25.404448 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:41:25.404415 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0407377_1192_4591_9515_1f6b9c40a9c2.slice/crio-b0c2be55cb2f790e4677808f9e15674ac9e82de7432945fe88c2ef34efdb5efe WatchSource:0}: Error finding container b0c2be55cb2f790e4677808f9e15674ac9e82de7432945fe88c2ef34efdb5efe: Status 404 returned error can't find the container with id b0c2be55cb2f790e4677808f9e15674ac9e82de7432945fe88c2ef34efdb5efe Oct 30 15:41:26.077136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:26.077060 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" event={"ID":"c0407377-1192-4591-9515-1f6b9c40a9c2","Type":"ContainerStarted","Data":"34481886cd6802d661919e24c2243c8ba532cef55a99a28f9c69684180e4e9e1"} Oct 30 15:41:26.077521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:26.077148 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" event={"ID":"c0407377-1192-4591-9515-1f6b9c40a9c2","Type":"ContainerStarted","Data":"b0c2be55cb2f790e4677808f9e15674ac9e82de7432945fe88c2ef34efdb5efe"} Oct 30 15:41:26.121417 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:26.121374 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" podStartSLOduration=2.12134535 podStartE2EDuration="2.12134535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:41:26.120145893 +0000 UTC m=+3190.079187300" watchObservedRunningTime="2026-10-30 15:41:26.12134535 +0000 UTC m=+3190.080386769" Oct 30 15:41:27.083276 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:27.083241 6554 generic.go:356] "Generic (PLEG): container finished" podID="c0407377-1192-4591-9515-1f6b9c40a9c2" containerID="34481886cd6802d661919e24c2243c8ba532cef55a99a28f9c69684180e4e9e1" exitCode=0 Oct 30 15:41:27.083692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:27.083326 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" event={"ID":"c0407377-1192-4591-9515-1f6b9c40a9c2","Type":"ContainerDied","Data":"34481886cd6802d661919e24c2243c8ba532cef55a99a28f9c69684180e4e9e1"} Oct 30 15:41:28.162854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.162798 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerName="payload-processing" containerID="cri-o://57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134" gracePeriod=25 Oct 30 15:41:28.202455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.202416 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerName="payload-pre-processing" containerID="cri-o://361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba" gracePeriod=25 Oct 30 15:41:28.240791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.240762 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:28.328614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.328589 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca\") pod \"c0407377-1192-4591-9515-1f6b9c40a9c2\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " Oct 30 15:41:28.328746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.328708 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c\" (UniqueName: \"kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c\") pod \"c0407377-1192-4591-9515-1f6b9c40a9c2\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " Oct 30 15:41:28.328804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.328771 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca\") pod \"c0407377-1192-4591-9515-1f6b9c40a9c2\" (UID: \"c0407377-1192-4591-9515-1f6b9c40a9c2\") " Oct 30 15:41:28.328941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.328915 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca" pod "c0407377-1192-4591-9515-1f6b9c40a9c2" (UID: "c0407377-1192-4591-9515-1f6b9c40a9c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:28.329240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.329221 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.330794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.330772 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca" pod "c0407377-1192-4591-9515-1f6b9c40a9c2" (UID: "c0407377-1192-4591-9515-1f6b9c40a9c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:28.330880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.330862 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c" pod "c0407377-1192-4591-9515-1f6b9c40a9c2" (UID: "c0407377-1192-4591-9515-1f6b9c40a9c2"). InnerVolumeSpecName "kube-api-access-gzs8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:28.430401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.430377 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzs8c\" (UniqueName: \"kubernetes.io/projected/c0407377-1192-4591-9515-1f6b9c40a9c2-kube-api-access-gzs8c\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.430401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.430402 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0407377-1192-4591-9515-1f6b9c40a9c2-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.573450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.573425 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 15:41:28.631386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631300 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631386 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631335 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631413 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631470 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631487 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631564 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631608 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631669 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631735 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert\") pod \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\" (UID: \"b980b704-8bd8-46ce-b9ee-de7ddc035ec5\") " Oct 30 15:41:28.631899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.631824 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:28.632069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632001 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:28.632069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632051 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:28.632245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632223 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:28.632451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632431 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.632659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632628 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.632659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632653 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.632818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632669 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.632818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.632444 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:28.633948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.633922 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:41:28.634079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.634063 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:28.634596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.634576 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:28.634670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.634603 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9" pod "b980b704-8bd8-46ce-b9ee-de7ddc035ec5" (UID: "b980b704-8bd8-46ce-b9ee-de7ddc035ec5"). InnerVolumeSpecName "kube-api-access-v9cg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:28.733552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.733528 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cg9\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-kube-api-access-v9cg9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.733552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.733552 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.733695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.733561 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.733695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.733571 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:28.733695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:28.733580 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b980b704-8bd8-46ce-b9ee-de7ddc035ec5-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.101047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.100869 6554 generic.go:356] "Generic (PLEG): container finished" podID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerID="aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f" exitCode=0 Oct 30 15:41:29.101047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.100934 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" Oct 30 15:41:29.101047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.100920 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" event={"ID":"b980b704-8bd8-46ce-b9ee-de7ddc035ec5","Type":"ContainerDied","Data":"aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f"} Oct 30 15:41:29.101047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.101018 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9" event={"ID":"b980b704-8bd8-46ce-b9ee-de7ddc035ec5","Type":"ContainerDied","Data":"5bf72c3fad2d46a3cc7c3205c17b9f7d1c2c4ce1d6c6f2dc6b0867ab1c2ca313"} Oct 30 15:41:29.101047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.101054 6554 scope.go:122] "RemoveContainer" containerID="aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f" Oct 30 15:41:29.102910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.102880 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" event={"ID":"c0407377-1192-4591-9515-1f6b9c40a9c2","Type":"ContainerDied","Data":"b0c2be55cb2f790e4677808f9e15674ac9e82de7432945fe88c2ef34efdb5efe"} Oct 30 15:41:29.102910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.102891 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g" Oct 30 15:41:29.103050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.102922 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0c2be55cb2f790e4677808f9e15674ac9e82de7432945fe88c2ef34efdb5efe" Oct 30 15:41:29.110977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.110934 6554 scope.go:122] "RemoveContainer" containerID="1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63" Oct 30 15:41:29.121422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.121403 6554 scope.go:122] "RemoveContainer" containerID="aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f" Oct 30 15:41:29.121745 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:29.121723 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f\": container with ID starting with aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f not found: ID does not exist" containerID="aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f" Oct 30 15:41:29.121801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.121754 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f"} err="failed to get container status \"aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f\": rpc error: code = NotFound desc = could not find container \"aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f\": container with ID starting with aeb290d1517b8791fb03b827864d6626159adbe1647db317c10f56d3b71e615f not found: ID does not exist" Oct 30 15:41:29.121801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.121771 6554 scope.go:122] "RemoveContainer" containerID="1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63" Oct 30 15:41:29.122041 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:29.122025 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63\": container with ID starting with 1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63 not found: ID does not exist" containerID="1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63" Oct 30 15:41:29.122086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.122046 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63"} err="failed to get container status \"1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63\": rpc error: code = NotFound desc = could not find container \"1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63\": container with ID starting with 1491bdd1903da260cf21dc67ab56b5fdd8210f23f8ca4743aa012b907ba78d63 not found: ID does not exist" Oct 30 15:41:29.178408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.177575 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9"] Oct 30 15:41:29.183439 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.183410 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-357948-openshift-default-6ffb96ccdc-nqln9"] Oct 30 15:41:29.246823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.246772 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:41:29.339197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339172 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339213 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339249 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339275 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339296 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339382 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339413 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339441 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339494 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket\") pod \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\" (UID: \"1a67a0ab-124f-412e-b5ee-b4bdadb5613d\") " Oct 30 15:41:29.339836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339686 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:29.339836 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339698 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:29.339948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339853 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:29.339948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339872 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:29.339948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339933 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:29.340110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339986 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.340110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.339998 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.340110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.340007 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.340110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.340024 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.341505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.341480 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:29.341692 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.341669 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:29.341813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.341793 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:41:29.341877 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.341798 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2" pod "1a67a0ab-124f-412e-b5ee-b4bdadb5613d" (UID: "1a67a0ab-124f-412e-b5ee-b4bdadb5613d"). InnerVolumeSpecName "kube-api-access-gd8z2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:29.441443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.441416 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.441443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.441440 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd8z2\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-kube-api-access-gd8z2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.441571 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.441450 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.441571 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.441458 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:29.441571 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:29.441466 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a67a0ab-124f-412e-b5ee-b4bdadb5613d-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:30.112675 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.112640 6554 generic.go:356] "Generic (PLEG): container finished" podID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerID="698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30" exitCode=0 Oct 30 15:41:30.112882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.112688 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" Oct 30 15:41:30.112882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.112724 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" event={"ID":"1a67a0ab-124f-412e-b5ee-b4bdadb5613d","Type":"ContainerDied","Data":"698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30"} Oct 30 15:41:30.112882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.112774 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr" event={"ID":"1a67a0ab-124f-412e-b5ee-b4bdadb5613d","Type":"ContainerDied","Data":"996a94d1ec3ed4e42c8f9803348dc9176f96f3496de366dd6d988a528af0bc6b"} Oct 30 15:41:30.112882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.112797 6554 scope.go:122] "RemoveContainer" containerID="698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30" Oct 30 15:41:30.122425 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.122409 6554 scope.go:122] "RemoveContainer" containerID="b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f" Oct 30 15:41:30.132527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.132509 6554 scope.go:122] "RemoveContainer" containerID="698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30" Oct 30 15:41:30.132780 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:30.132761 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30\": container with ID starting with 698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30 not found: ID does not exist" containerID="698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30" Oct 30 15:41:30.132861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.132791 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30"} err="failed to get container status \"698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30\": rpc error: code = NotFound desc = could not find container \"698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30\": container with ID starting with 698b0c035a80ac78e6d25b22724dad5afea3569276c133153f49be5bc24f1d30 not found: ID does not exist" Oct 30 15:41:30.132861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.132815 6554 scope.go:122] "RemoveContainer" containerID="b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f" Oct 30 15:41:30.133056 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:30.133040 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f\": container with ID starting with b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f not found: ID does not exist" containerID="b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f" Oct 30 15:41:30.133098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.133061 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f"} err="failed to get container status \"b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f\": rpc error: code = NotFound desc = could not find container \"b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f\": container with ID starting with b822016f89d6c0581e2db2c89f5753908dd68541053ec47e338be2c52392271f not found: ID does not exist" Oct 30 15:41:30.153283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.153263 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" path="/var/lib/kubelet/pods/b980b704-8bd8-46ce-b9ee-de7ddc035ec5/volumes" Oct 30 15:41:30.182966 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.182943 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr"] Oct 30 15:41:30.197325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:30.197300 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-5a5948-openshift-default-57c76dd9df-dj6vr"] Oct 30 15:41:32.155392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:32.155340 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" path="/var/lib/kubelet/pods/1a67a0ab-124f-412e-b5ee-b4bdadb5613d/volumes" Oct 30 15:41:34.202929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:34.202894 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:34.203450 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:34.202905 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:35.071297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:35.071263 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc"] Oct 30 15:41:35.104795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:35.104765 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m"] Oct 30 15:41:39.201046 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:39.201010 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:41:40.071774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.071739 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" podUID="f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" containerName="payload-processing" containerID="cri-o://5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b" gracePeriod=25 Oct 30 15:41:40.105717 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.105668 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" podUID="3571d646-2915-4c39-8db1-e35152a7027d" containerName="payload-pre-processing" containerID="cri-o://4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc" gracePeriod=25 Oct 30 15:41:40.367839 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.367802 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 15:41:40.403125 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.403100 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 15:41:40.445569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.445534 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb\" (UniqueName: \"kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb\") pod \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " Oct 30 15:41:40.445968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.445705 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume\") pod \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\" (UID: \"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5\") " Oct 30 15:41:40.446345 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.446317 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume" pod "f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" (UID: "f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:40.446585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.446563 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:40.449169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.449147 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb" pod "f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" (UID: "f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5"). InnerVolumeSpecName "kube-api-access-brbqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:40.547900 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.547876 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume\") pod \"3571d646-2915-4c39-8db1-e35152a7027d\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " Oct 30 15:41:40.548009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.547918 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9\" (UniqueName: \"kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9\") pod \"3571d646-2915-4c39-8db1-e35152a7027d\" (UID: \"3571d646-2915-4c39-8db1-e35152a7027d\") " Oct 30 15:41:40.548160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.548147 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brbqb\" (UniqueName: \"kubernetes.io/projected/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5-kube-api-access-brbqb\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:40.548309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.548282 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume" pod "3571d646-2915-4c39-8db1-e35152a7027d" (UID: "3571d646-2915-4c39-8db1-e35152a7027d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:40.550003 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.549982 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9" pod "3571d646-2915-4c39-8db1-e35152a7027d" (UID: "3571d646-2915-4c39-8db1-e35152a7027d"). InnerVolumeSpecName "kube-api-access-rrzb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:40.649609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.649570 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3571d646-2915-4c39-8db1-e35152a7027d-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:40.649609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:40.649604 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzb9\" (UniqueName: \"kubernetes.io/projected/3571d646-2915-4c39-8db1-e35152a7027d-kube-api-access-rrzb9\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:41.172638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.172607 6554 generic.go:356] "Generic (PLEG): container finished" podID="f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" containerID="5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b" exitCode=0 Oct 30 15:41:41.172769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.172653 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" Oct 30 15:41:41.172769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.172690 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" event={"ID":"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5","Type":"ContainerDied","Data":"5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b"} Oct 30 15:41:41.172769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.172740 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc" event={"ID":"f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5","Type":"ContainerDied","Data":"27447ee701a8428fc1d27c2fce3048d646e1dc7b91adbec6f985bc917080978e"} Oct 30 15:41:41.172769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.172768 6554 scope.go:122] "RemoveContainer" containerID="5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b" Oct 30 15:41:41.174194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.174175 6554 generic.go:356] "Generic (PLEG): container finished" podID="3571d646-2915-4c39-8db1-e35152a7027d" containerID="4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc" exitCode=0 Oct 30 15:41:41.174292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.174214 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" event={"ID":"3571d646-2915-4c39-8db1-e35152a7027d","Type":"ContainerDied","Data":"4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc"} Oct 30 15:41:41.174292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.174228 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" Oct 30 15:41:41.174292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.174250 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m" event={"ID":"3571d646-2915-4c39-8db1-e35152a7027d","Type":"ContainerDied","Data":"05593a128dbd6e4663ecc3f797d95d30491ef05cf741900c9d8d45ac334e9d0f"} Oct 30 15:41:41.184601 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.184582 6554 scope.go:122] "RemoveContainer" containerID="5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b" Oct 30 15:41:41.184893 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:41.184872 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b\": container with ID starting with 5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b not found: ID does not exist" containerID="5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b" Oct 30 15:41:41.184970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.184899 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b"} err="failed to get container status \"5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b\": rpc error: code = NotFound desc = could not find container \"5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b\": container with ID starting with 5b2e06071bfbf8fbe62f6722c1a43a9b2dc66765311b223311c9502f4433261b not found: ID does not exist" Oct 30 15:41:41.184970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.184916 6554 scope.go:122] "RemoveContainer" containerID="4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc" Oct 30 15:41:41.193045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.193028 6554 scope.go:122] "RemoveContainer" containerID="4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc" Oct 30 15:41:41.193329 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:41.193304 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc\": container with ID starting with 4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc not found: ID does not exist" containerID="4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc" Oct 30 15:41:41.193426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.193340 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc"} err="failed to get container status \"4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc\": rpc error: code = NotFound desc = could not find container \"4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc\": container with ID starting with 4ec238c944be3504d1343b6c4afbb291fa67cda568f985771d9550054f4586dc not found: ID does not exist" Oct 30 15:41:41.228573 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.228547 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m"] Oct 30 15:41:41.237635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.237611 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-4b703c-5d5c5d7d75-6xc8m"] Oct 30 15:41:41.274915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.274891 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc"] Oct 30 15:41:41.277876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:41.277856 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4b703c-bd655fc6d-2gxnc"] Oct 30 15:41:42.156303 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:42.156258 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3571d646-2915-4c39-8db1-e35152a7027d" path="/var/lib/kubelet/pods/3571d646-2915-4c39-8db1-e35152a7027d/volumes" Oct 30 15:41:42.165229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:42.157369 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" path="/var/lib/kubelet/pods/f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5/volumes" Oct 30 15:41:44.203598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:44.203568 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:44.203998 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:44.203623 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:41:44.770088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:44.770052 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz"] Oct 30 15:41:44.770361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:44.770282 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" containerID="cri-o://df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975" gracePeriod=30 Oct 30 15:41:47.551870 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:47.551832 6554 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:41:47.552384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:47.551895 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:41:48.062578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.062536 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs"] Oct 30 15:41:48.062784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.062749 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" containerID="cri-o://4c8e2cb350c535b4ec993c8a6854b8af26c8cca58bea5ec1af452a5cb02c2fb0" gracePeriod=30 Oct 30 15:41:48.459766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.459739 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 15:41:48.480016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.479994 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 15:41:48.624787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.624720 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume\") pod \"2e069241-0830-4dbe-8fba-f1e867496dbb\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " Oct 30 15:41:48.624787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.624756 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume\") pod \"f683f8ad-7e91-49a1-abe4-6b099388fc81\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " Oct 30 15:41:48.625190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.624806 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r\" (UniqueName: \"kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r\") pod \"f683f8ad-7e91-49a1-abe4-6b099388fc81\" (UID: \"f683f8ad-7e91-49a1-abe4-6b099388fc81\") " Oct 30 15:41:48.625190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.624908 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc\" (UniqueName: \"kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc\") pod \"2e069241-0830-4dbe-8fba-f1e867496dbb\" (UID: \"2e069241-0830-4dbe-8fba-f1e867496dbb\") " Oct 30 15:41:48.625190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.625140 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume" pod "2e069241-0830-4dbe-8fba-f1e867496dbb" (UID: "2e069241-0830-4dbe-8fba-f1e867496dbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:48.625315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.625188 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume" pod "f683f8ad-7e91-49a1-abe4-6b099388fc81" (UID: "f683f8ad-7e91-49a1-abe4-6b099388fc81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:48.625479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.625460 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e069241-0830-4dbe-8fba-f1e867496dbb-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:48.625522 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.625485 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f683f8ad-7e91-49a1-abe4-6b099388fc81-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:48.626903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.626882 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r" pod "f683f8ad-7e91-49a1-abe4-6b099388fc81" (UID: "f683f8ad-7e91-49a1-abe4-6b099388fc81"). InnerVolumeSpecName "kube-api-access-f6l4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:48.626988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.626973 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc" pod "2e069241-0830-4dbe-8fba-f1e867496dbb" (UID: "2e069241-0830-4dbe-8fba-f1e867496dbb"). InnerVolumeSpecName "kube-api-access-xglmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:48.725988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.725961 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6l4r\" (UniqueName: \"kubernetes.io/projected/f683f8ad-7e91-49a1-abe4-6b099388fc81-kube-api-access-f6l4r\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:48.725988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:48.725983 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xglmc\" (UniqueName: \"kubernetes.io/projected/2e069241-0830-4dbe-8fba-f1e867496dbb-kube-api-access-xglmc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:49.225548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.225510 6554 generic.go:356] "Generic (PLEG): container finished" podID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerID="361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba" exitCode=0 Oct 30 15:41:49.225716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.225574 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" Oct 30 15:41:49.225716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.225601 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" event={"ID":"f683f8ad-7e91-49a1-abe4-6b099388fc81","Type":"ContainerDied","Data":"361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba"} Oct 30 15:41:49.225716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.225685 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv" event={"ID":"f683f8ad-7e91-49a1-abe4-6b099388fc81","Type":"ContainerDied","Data":"d1c580919f7a4ab1e6a6ef736a6e0317b0505d24e81e2c66f602edced326f649"} Oct 30 15:41:49.225716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.225709 6554 scope.go:122] "RemoveContainer" containerID="361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba" Oct 30 15:41:49.227229 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.227203 6554 generic.go:356] "Generic (PLEG): container finished" podID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerID="57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134" exitCode=0 Oct 30 15:41:49.227342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.227240 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" event={"ID":"2e069241-0830-4dbe-8fba-f1e867496dbb","Type":"ContainerDied","Data":"57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134"} Oct 30 15:41:49.227342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.227254 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" Oct 30 15:41:49.227342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.227280 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn" event={"ID":"2e069241-0830-4dbe-8fba-f1e867496dbb","Type":"ContainerDied","Data":"d909860d3b6c595d47e43e4504f2ca74ef3477cf87ff6126da6dad372d620fe1"} Oct 30 15:41:49.236067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.235948 6554 scope.go:122] "RemoveContainer" containerID="361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba" Oct 30 15:41:49.236280 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:49.236260 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba\": container with ID starting with 361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba not found: ID does not exist" containerID="361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba" Oct 30 15:41:49.236367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.236285 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba"} err="failed to get container status \"361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba\": rpc error: code = NotFound desc = could not find container \"361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba\": container with ID starting with 361e0a1af7df0de57b6d3c9d3fc536ff3ed8743e88df0ae66fd671c229d8dbba not found: ID does not exist" Oct 30 15:41:49.236367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.236301 6554 scope.go:122] "RemoveContainer" containerID="57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134" Oct 30 15:41:49.244429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.244413 6554 scope.go:122] "RemoveContainer" containerID="57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134" Oct 30 15:41:49.244700 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:49.244683 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134\": container with ID starting with 57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134 not found: ID does not exist" containerID="57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134" Oct 30 15:41:49.244765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.244708 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134"} err="failed to get container status \"57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134\": rpc error: code = NotFound desc = could not find container \"57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134\": container with ID starting with 57e8d03d29197a641cc9171ddd104bc374d964ae9fa55969fa99f29353447134 not found: ID does not exist" Oct 30 15:41:49.280116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.280087 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn"] Oct 30 15:41:49.293554 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.293531 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e64e05-578b48874d-6q8nn"] Oct 30 15:41:49.321096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.321072 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv"] Oct 30 15:41:49.325493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.325473 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e64e05-6bb965bb9f-v5svv"] Oct 30 15:41:49.772997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.772865 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b"] Oct 30 15:41:49.773572 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:49.773099 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" podUID="30837e40-0881-430e-a9bb-4260ea32b5f6" containerName="maas-api" containerID="cri-o://01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d" gracePeriod=30 Oct 30 15:41:50.030103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.030040 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:41:50.069768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.069628 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 15:41:50.140831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.140793 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.140844 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.140896 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.140926 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.140971 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141074 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141144 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141176 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141204 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert\") pod \"53a97a21-862a-4156-9ffb-8dfadc5b306c\" (UID: \"53a97a21-862a-4156-9ffb-8dfadc5b306c\") " Oct 30 15:41:50.141499 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141376 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:50.141712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141691 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:50.141822 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141709 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.142208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.141954 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:50.142208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.142133 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:50.142208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.142152 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:50.143242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.143217 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:50.143605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.143586 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "kube-api-access-f5g5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:50.143824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.143796 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:41:50.143948 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.143847 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token" pod "53a97a21-862a-4156-9ffb-8dfadc5b306c" (UID: "53a97a21-862a-4156-9ffb-8dfadc5b306c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:50.153725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.153701 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" path="/var/lib/kubelet/pods/2e069241-0830-4dbe-8fba-f1e867496dbb/volumes" Oct 30 15:41:50.154131 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.154113 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" path="/var/lib/kubelet/pods/f683f8ad-7e91-49a1-abe4-6b099388fc81/volumes" Oct 30 15:41:50.234257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.234237 6554 generic.go:356] "Generic (PLEG): container finished" podID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerID="df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975" exitCode=0 Oct 30 15:41:50.234378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.234286 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" Oct 30 15:41:50.234378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.234313 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" event={"ID":"53a97a21-862a-4156-9ffb-8dfadc5b306c","Type":"ContainerDied","Data":"df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975"} Oct 30 15:41:50.234378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.234373 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz" event={"ID":"53a97a21-862a-4156-9ffb-8dfadc5b306c","Type":"ContainerDied","Data":"5bbaaefe0969c1778e28b32d148ac9e4ec55a8c6d9bf400794331e31bd8ff5d6"} Oct 30 15:41:50.234521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.234395 6554 scope.go:122] "RemoveContainer" containerID="df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975" Oct 30 15:41:50.235712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.235693 6554 generic.go:356] "Generic (PLEG): container finished" podID="30837e40-0881-430e-a9bb-4260ea32b5f6" containerID="01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d" exitCode=0 Oct 30 15:41:50.235791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.235734 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" Oct 30 15:41:50.235855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.235781 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" event={"ID":"30837e40-0881-430e-a9bb-4260ea32b5f6","Type":"ContainerDied","Data":"01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d"} Oct 30 15:41:50.235919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.235854 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b" event={"ID":"30837e40-0881-430e-a9bb-4260ea32b5f6","Type":"ContainerDied","Data":"1f670801d8b7a3bebe2435b1202e7179f369af7f60e89721354b65d2373f2324"} Oct 30 15:41:50.244686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.244663 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2\" (UniqueName: \"kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2\") pod \"30837e40-0881-430e-a9bb-4260ea32b5f6\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.244729 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls\") pod \"30837e40-0881-430e-a9bb-4260ea32b5f6\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.244815 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca\") pod \"30837e40-0881-430e-a9bb-4260ea32b5f6\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.244848 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs\") pod \"30837e40-0881-430e-a9bb-4260ea32b5f6\" (UID: \"30837e40-0881-430e-a9bb-4260ea32b5f6\") " Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245264 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca" pod "30837e40-0881-430e-a9bb-4260ea32b5f6" (UID: "30837e40-0881-430e-a9bb-4260ea32b5f6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245766 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245789 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245803 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245816 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53a97a21-862a-4156-9ffb-8dfadc5b306c-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245831 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30837e40-0881-430e-a9bb-4260ea32b5f6-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245844 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245856 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245868 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5g5z\" (UniqueName: \"kubernetes.io/projected/53a97a21-862a-4156-9ffb-8dfadc5b306c-kube-api-access-f5g5z\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.246405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.245883 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53a97a21-862a-4156-9ffb-8dfadc5b306c-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.247679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.247643 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs" pod "30837e40-0881-430e-a9bb-4260ea32b5f6" (UID: "30837e40-0881-430e-a9bb-4260ea32b5f6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:50.248114 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.247973 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls" pod "30837e40-0881-430e-a9bb-4260ea32b5f6" (UID: "30837e40-0881-430e-a9bb-4260ea32b5f6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:50.248196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.248111 6554 scope.go:122] "RemoveContainer" containerID="646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2" Oct 30 15:41:50.249540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.249128 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2" pod "30837e40-0881-430e-a9bb-4260ea32b5f6" (UID: "30837e40-0881-430e-a9bb-4260ea32b5f6"). InnerVolumeSpecName "kube-api-access-gbkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:50.270206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.270181 6554 scope.go:122] "RemoveContainer" containerID="df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975" Oct 30 15:41:50.270672 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:50.270649 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975\": container with ID starting with df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975 not found: ID does not exist" containerID="df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975" Oct 30 15:41:50.270785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.270681 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975"} err="failed to get container status \"df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975\": rpc error: code = NotFound desc = could not find container \"df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975\": container with ID starting with df12ce8377bfac5ad432d32dd290c06bd35678f8ff57a93d063061c4d7076975 not found: ID does not exist" Oct 30 15:41:50.270785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.270696 6554 scope.go:122] "RemoveContainer" containerID="646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2" Oct 30 15:41:50.271040 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:50.270971 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2\": container with ID starting with 646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2 not found: ID does not exist" containerID="646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2" Oct 30 15:41:50.271040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.271012 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2"} err="failed to get container status \"646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2\": rpc error: code = NotFound desc = could not find container \"646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2\": container with ID starting with 646b4aa216d32f5bda04f03986a2dbbc3f10bb788ce29c60bf996a3a7f9a60b2 not found: ID does not exist" Oct 30 15:41:50.271040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.271030 6554 scope.go:122] "RemoveContainer" containerID="01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d" Oct 30 15:41:50.280029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.279997 6554 scope.go:122] "RemoveContainer" containerID="01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d" Oct 30 15:41:50.280431 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:50.280322 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d\": container with ID starting with 01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d not found: ID does not exist" containerID="01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d" Oct 30 15:41:50.280431 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.280368 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d"} err="failed to get container status \"01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d\": rpc error: code = NotFound desc = could not find container \"01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d\": container with ID starting with 01b0cad1402c7c3a1d658bf731e820a92767c567a977961d52c4f419cae3859d not found: ID does not exist" Oct 30 15:41:50.290539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.290506 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz"] Oct 30 15:41:50.294048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.294027 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b236a6-openshift-default-75f495bb77-4cjgz"] Oct 30 15:41:50.346393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.346371 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkq2\" (UniqueName: \"kubernetes.io/projected/30837e40-0881-430e-a9bb-4260ea32b5f6-kube-api-access-gbkq2\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.346498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.346396 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.346498 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.346412 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30837e40-0881-430e-a9bb-4260ea32b5f6-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:50.583741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.583718 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b"] Oct 30 15:41:50.589226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:50.589205 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-4b703c-fc79dbd67-29t9b"] Oct 30 15:41:52.153688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:52.153654 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30837e40-0881-430e-a9bb-4260ea32b5f6" path="/var/lib/kubelet/pods/30837e40-0881-430e-a9bb-4260ea32b5f6/volumes" Oct 30 15:41:52.154087 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:52.154073 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" path="/var/lib/kubelet/pods/53a97a21-862a-4156-9ffb-8dfadc5b306c/volumes" Oct 30 15:41:53.265564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.265530 6554 generic.go:356] "Generic (PLEG): container finished" podID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerID="4c8e2cb350c535b4ec993c8a6854b8af26c8cca58bea5ec1af452a5cb02c2fb0" exitCode=0 Oct 30 15:41:53.265925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.265579 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" event={"ID":"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad","Type":"ContainerDied","Data":"4c8e2cb350c535b4ec993c8a6854b8af26c8cca58bea5ec1af452a5cb02c2fb0"} Oct 30 15:41:53.318511 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.318485 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:41:53.474857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.474830 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.474874 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.474897 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.474942 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.474979 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475005 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475056 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475075 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475129 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:53.475330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475137 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token\") pod \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\" (UID: \"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad\") " Oct 30 15:41:53.475330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475293 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:53.475666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475328 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:53.475666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475526 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:53.475666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475540 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:53.475844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475698 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.475844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475713 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.475844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475723 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.475844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475732 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.475844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.475739 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.477364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.477331 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:41:53.477471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.477342 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:41:53.477666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.477644 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:53.477785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.477703 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k" pod "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" (UID: "6e3f5a2c-88f7-49ef-90fd-f37dbae240ad"). InnerVolumeSpecName "kube-api-access-8sv2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:53.576995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.576972 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sv2k\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-kube-api-access-8sv2k\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.576995 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.576993 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.577128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.577004 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:53.577128 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:53.577017 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:54.272695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:54.272669 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" Oct 30 15:41:54.273205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:54.272740 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs" event={"ID":"6e3f5a2c-88f7-49ef-90fd-f37dbae240ad","Type":"ContainerDied","Data":"efbe226ce684a32384526b62717da8c55138e9a4ba52d2c0888880798733ddae"} Oct 30 15:41:54.273205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:54.272797 6554 scope.go:122] "RemoveContainer" containerID="4c8e2cb350c535b4ec993c8a6854b8af26c8cca58bea5ec1af452a5cb02c2fb0" Oct 30 15:41:54.345680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:54.345651 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs"] Oct 30 15:41:54.358946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:54.358926 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e64e05-openshift-default-5d86d57c55-s8lfs"] Oct 30 15:41:55.713858 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:55.713821 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9"] Oct 30 15:41:55.714309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:55.714032 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" podUID="827f20c2-e338-4eee-b752-a36145cee6ba" containerName="maas-api" containerID="cri-o://879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061" gracePeriod=30 Oct 30 15:41:55.803329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:55.803296 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc"] Oct 30 15:41:55.835391 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:55.835362 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9"] Oct 30 15:41:55.987755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:55.987727 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:41:56.098869 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.098843 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca\") pod \"827f20c2-e338-4eee-b752-a36145cee6ba\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " Oct 30 15:41:56.099026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.098881 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\") pod \"827f20c2-e338-4eee-b752-a36145cee6ba\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " Oct 30 15:41:56.099026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.098924 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp\" (UniqueName: \"kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp\") pod \"827f20c2-e338-4eee-b752-a36145cee6ba\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " Oct 30 15:41:56.099116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.099024 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs\") pod \"827f20c2-e338-4eee-b752-a36145cee6ba\" (UID: \"827f20c2-e338-4eee-b752-a36145cee6ba\") " Oct 30 15:41:56.099271 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.099240 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca" pod "827f20c2-e338-4eee-b752-a36145cee6ba" (UID: "827f20c2-e338-4eee-b752-a36145cee6ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:41:56.101206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.101182 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls" pod "827f20c2-e338-4eee-b752-a36145cee6ba" (UID: "827f20c2-e338-4eee-b752-a36145cee6ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:56.101336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.101319 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp" pod "827f20c2-e338-4eee-b752-a36145cee6ba" (UID: "827f20c2-e338-4eee-b752-a36145cee6ba"). InnerVolumeSpecName "kube-api-access-dg6qp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:41:56.101410 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.101328 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs" pod "827f20c2-e338-4eee-b752-a36145cee6ba" (UID: "827f20c2-e338-4eee-b752-a36145cee6ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:41:56.153080 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.153055 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" path="/var/lib/kubelet/pods/6e3f5a2c-88f7-49ef-90fd-f37dbae240ad/volumes" Oct 30 15:41:56.199985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.199953 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/827f20c2-e338-4eee-b752-a36145cee6ba-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:56.199985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.199981 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:56.199985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.199991 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg6qp\" (UniqueName: \"kubernetes.io/projected/827f20c2-e338-4eee-b752-a36145cee6ba-kube-api-access-dg6qp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:56.200320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.200000 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/827f20c2-e338-4eee-b752-a36145cee6ba-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:41:56.287120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.287045 6554 generic.go:356] "Generic (PLEG): container finished" podID="827f20c2-e338-4eee-b752-a36145cee6ba" containerID="879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061" exitCode=0 Oct 30 15:41:56.287120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.287093 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" Oct 30 15:41:56.287312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.287122 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" event={"ID":"827f20c2-e338-4eee-b752-a36145cee6ba","Type":"ContainerDied","Data":"879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061"} Oct 30 15:41:56.287312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.287188 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9" event={"ID":"827f20c2-e338-4eee-b752-a36145cee6ba","Type":"ContainerDied","Data":"d6ee2083d4122d34c166704c151950f6e4b0a9af22eab39971432965af3b33ed"} Oct 30 15:41:56.287312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.287213 6554 scope.go:122] "RemoveContainer" containerID="879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061" Oct 30 15:41:56.296130 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.296114 6554 scope.go:122] "RemoveContainer" containerID="879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061" Oct 30 15:41:56.296394 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:41:56.296371 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061\": container with ID starting with 879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061 not found: ID does not exist" containerID="879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061" Oct 30 15:41:56.296496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.296399 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061"} err="failed to get container status \"879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061\": rpc error: code = NotFound desc = could not find container \"879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061\": container with ID starting with 879cb409b23a02406f3d0e823ea7e4902e8f39080edd4fbbcf29fc4d0f382061 not found: ID does not exist" Oct 30 15:41:56.343916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.343895 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9"] Oct 30 15:41:56.349048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:56.349028 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-a1654b-8648c9788-jjxk9"] Oct 30 15:41:58.153495 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:58.153462 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827f20c2-e338-4eee-b752-a36145cee6ba" path="/var/lib/kubelet/pods/827f20c2-e338-4eee-b752-a36145cee6ba/volumes" Oct 30 15:41:58.302110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:58.302077 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc"] Oct 30 15:41:58.302387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:41:58.302330 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" containerID="cri-o://255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485" gracePeriod=30 Oct 30 15:42:00.803605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:00.803547 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerName="payload-processing" containerID="cri-o://379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0" gracePeriod=25 Oct 30 15:42:00.835800 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:00.835764 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerName="payload-pre-processing" containerID="cri-o://20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557" gracePeriod=25 Oct 30 15:42:01.851211 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:01.851177 6554 patch_prober.go:29] interesting pod/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 15:42:01.851627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:01.851230 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 15:42:03.548664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.548639 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:42:03.667167 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667141 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667191 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667220 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667267 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667292 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667414 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667471 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667502 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667559 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy\") pod \"a33c93d8-f005-4910-b132-f02662ec3596\" (UID: \"a33c93d8-f005-4910-b132-f02662ec3596\") " Oct 30 15:42:03.667706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667608 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:03.667706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667694 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:03.667890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667825 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:03.667972 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667949 6554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-data\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.668035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667979 6554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-credential-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.668035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.667994 6554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-socket\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.668035 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.668006 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:03.668202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.668084 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:03.669475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.669448 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "kube-api-access-fzbdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:03.669623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.669596 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 15:42:03.669698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.669680 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:03.669873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.669856 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token" pod "a33c93d8-f005-4910-b132-f02662ec3596" (UID: "a33c93d8-f005-4910-b132-f02662ec3596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:03.768605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.768575 6554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-istio-envoy\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.768605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.768606 6554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-istio-token\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.768837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.768620 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzbdp\" (UniqueName: \"kubernetes.io/projected/a33c93d8-f005-4910-b132-f02662ec3596-kube-api-access-fzbdp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.768837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.768633 6554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a33c93d8-f005-4910-b132-f02662ec3596-istio-podinfo\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.768837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.768645 6554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a33c93d8-f005-4910-b132-f02662ec3596-istiod-ca-cert\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:03.768837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:03.768658 6554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33c93d8-f005-4910-b132-f02662ec3596-workload-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:04.329316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.329279 6554 generic.go:356] "Generic (PLEG): container finished" podID="a33c93d8-f005-4910-b132-f02662ec3596" containerID="255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485" exitCode=0 Oct 30 15:42:04.329486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.329316 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" Oct 30 15:42:04.329486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.329326 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" event={"ID":"a33c93d8-f005-4910-b132-f02662ec3596","Type":"ContainerDied","Data":"255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485"} Oct 30 15:42:04.329486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.329402 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc" event={"ID":"a33c93d8-f005-4910-b132-f02662ec3596","Type":"ContainerDied","Data":"4e1a815979e343445c08d7f1f8718ba7c5711a2f8804775576ca046fcb658d87"} Oct 30 15:42:04.329486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.329427 6554 scope.go:122] "RemoveContainer" containerID="255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485" Oct 30 15:42:04.338636 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.338620 6554 scope.go:122] "RemoveContainer" containerID="255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485" Oct 30 15:42:04.338891 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:04.338871 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485\": container with ID starting with 255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485 not found: ID does not exist" containerID="255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485" Oct 30 15:42:04.338939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.338898 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485"} err="failed to get container status \"255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485\": rpc error: code = NotFound desc = could not find container \"255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485\": container with ID starting with 255b0e191789c6191099cd258ad91481743ce4f6960e6001ba0fd84be25e8485 not found: ID does not exist" Oct 30 15:42:04.398867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.398844 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc"] Oct 30 15:42:04.404557 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:04.404538 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4b703c-openshift-default-79b7cd57b7-qrbxc"] Oct 30 15:42:05.498139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498109 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv"] Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498508 6554 cpu_manager.go:414] "containerMap: removing container" podUID="827f20c2-e338-4eee-b752-a36145cee6ba" containerName="maas-api" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498520 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827f20c2-e338-4eee-b752-a36145cee6ba" containerName="maas-api" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498537 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498542 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498550 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498555 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498561 6554 cpu_manager.go:414] "containerMap: removing container" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498566 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498572 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" containerName="payload-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498577 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" containerName="payload-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498585 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3571d646-2915-4c39-8db1-e35152a7027d" containerName="payload-pre-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498589 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3571d646-2915-4c39-8db1-e35152a7027d" containerName="payload-pre-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498597 6554 cpu_manager.go:414] "containerMap: removing container" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerName="payload-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498602 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerName="payload-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498608 6554 cpu_manager.go:414] "containerMap: removing container" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerName="payload-pre-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498613 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerName="payload-pre-processing" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498619 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c0407377-1192-4591-9515-1f6b9c40a9c2" containerName="revoke-keys" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498623 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0407377-1192-4591-9515-1f6b9c40a9c2" containerName="revoke-keys" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498633 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" Oct 30 15:42:05.498633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498637 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498651 6554 cpu_manager.go:414] "containerMap: removing container" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498656 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498665 6554 cpu_manager.go:414] "containerMap: removing container" podUID="30837e40-0881-430e-a9bb-4260ea32b5f6" containerName="maas-api" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498670 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30837e40-0881-430e-a9bb-4260ea32b5f6" containerName="maas-api" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498678 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498683 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498689 6554 cpu_manager.go:414] "containerMap: removing container" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498693 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498703 6554 cpu_manager.go:414] "containerMap: removing container" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498707 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498757 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0407377-1192-4591-9515-1f6b9c40a9c2" containerName="revoke-keys" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498766 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a33c93d8-f005-4910-b132-f02662ec3596" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498773 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498780 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="30837e40-0881-430e-a9bb-4260ea32b5f6" containerName="maas-api" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498786 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e58e5fda-c858-4072-8631-307319269c5e" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498791 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498796 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="53a97a21-862a-4156-9ffb-8dfadc5b306c" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498802 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e3f5a2c-88f7-49ef-90fd-f37dbae240ad" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498809 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="827f20c2-e338-4eee-b752-a36145cee6ba" containerName="maas-api" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498815 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498821 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="b980b704-8bd8-46ce-b9ee-de7ddc035ec5" containerName="istio-proxy" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498826 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f767f3a1-2ee7-47eb-bb0d-d634b0f3ffa5" containerName="payload-processing" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498831 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3571d646-2915-4c39-8db1-e35152a7027d" containerName="payload-pre-processing" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498836 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e069241-0830-4dbe-8fba-f1e867496dbb" containerName="payload-processing" Oct 30 15:42:05.499379 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.498842 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="f683f8ad-7e91-49a1-abe4-6b099388fc81" containerName="payload-pre-processing" Oct 30 15:42:05.502769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.502751 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.522810 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.522786 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv"] Oct 30 15:42:05.535005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.534971 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:42:05.535005 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.534996 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:42:05.542516 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.542496 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:42:05.542768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.542750 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:42:05.684570 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.684546 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.684697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.684597 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkgd8\" (UniqueName: \"kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.684774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.684686 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.785687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.785634 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkgd8\" (UniqueName: \"kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.785687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.785668 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.785873 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.785732 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.786444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.786425 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.787976 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.787955 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.810217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.810194 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkgd8\" (UniqueName: \"kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8\") pod \"maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:05.814962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:05.814945 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:06.113251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:06.113222 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv"] Oct 30 15:42:06.114907 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:42:06.114877 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e0b2ba7_e03e_4a29_868d_cf8361cb3c2a.slice/crio-1bfafd80f3504a78251894b73e354bf08ae36f1d0eb58bea0b2c3a0d5da46226 WatchSource:0}: Error finding container 1bfafd80f3504a78251894b73e354bf08ae36f1d0eb58bea0b2c3a0d5da46226: Status 404 returned error can't find the container with id 1bfafd80f3504a78251894b73e354bf08ae36f1d0eb58bea0b2c3a0d5da46226 Oct 30 15:42:06.155067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:06.155039 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a33c93d8-f005-4910-b132-f02662ec3596" path="/var/lib/kubelet/pods/a33c93d8-f005-4910-b132-f02662ec3596/volumes" Oct 30 15:42:06.342109 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:06.342077 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" event={"ID":"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a","Type":"ContainerStarted","Data":"47380c6497399dbfc931fd24947694f1f3834c58924f2790f38b6b16d1b6eacc"} Oct 30 15:42:06.342239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:06.342130 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" event={"ID":"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a","Type":"ContainerStarted","Data":"1bfafd80f3504a78251894b73e354bf08ae36f1d0eb58bea0b2c3a0d5da46226"} Oct 30 15:42:06.386564 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:06.386519 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" podStartSLOduration=1.386505766 podStartE2EDuration="1.386505766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:06.385422225 +0000 UTC m=+3230.344463629" watchObservedRunningTime="2026-10-30 15:42:06.386505766 +0000 UTC m=+3230.345547200" Oct 30 15:42:07.348297 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:07.348265 6554 generic.go:356] "Generic (PLEG): container finished" podID="3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" containerID="47380c6497399dbfc931fd24947694f1f3834c58924f2790f38b6b16d1b6eacc" exitCode=0 Oct 30 15:42:07.348757 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:07.348339 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" event={"ID":"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a","Type":"ContainerDied","Data":"47380c6497399dbfc931fd24947694f1f3834c58924f2790f38b6b16d1b6eacc"} Oct 30 15:42:08.484714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.484689 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:08.610366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.610284 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8\" (UniqueName: \"kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8\") pod \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " Oct 30 15:42:08.610366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.610319 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca\") pod \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " Oct 30 15:42:08.610605 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.610388 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca\") pod \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\" (UID: \"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a\") " Oct 30 15:42:08.610786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.610765 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca" pod "3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" (UID: "3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:08.612385 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.612342 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8" pod "3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" (UID: "3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a"). InnerVolumeSpecName "kube-api-access-gkgd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:08.612484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.612437 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca" pod "3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" (UID: "3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:42:08.711735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.711710 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkgd8\" (UniqueName: \"kubernetes.io/projected/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-kube-api-access-gkgd8\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:08.711735 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.711732 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:08.711868 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:08.711742 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:09.200200 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:09.200164 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:42:09.358834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:09.358804 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" Oct 30 15:42:09.358834 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:09.358825 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv" event={"ID":"3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a","Type":"ContainerDied","Data":"1bfafd80f3504a78251894b73e354bf08ae36f1d0eb58bea0b2c3a0d5da46226"} Oct 30 15:42:09.359039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:09.358871 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bfafd80f3504a78251894b73e354bf08ae36f1d0eb58bea0b2c3a0d5da46226" Oct 30 15:42:09.897521 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:09.897486 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:09.897921 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:09.897527 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:10.307813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:10.307743 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg"] Oct 30 15:42:15.939101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:15.939064 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts"] Oct 30 15:42:15.974359 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:15.974316 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x"] Oct 30 15:42:16.004210 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.004184 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr"] Oct 30 15:42:16.004586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.004572 6554 cpu_manager.go:414] "containerMap: removing container" podUID="3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" containerName="revoke-keys" Oct 30 15:42:16.004634 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.004586 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" containerName="revoke-keys" Oct 30 15:42:16.004669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.004661 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a67a0ab-124f-412e-b5ee-b4bdadb5613d" containerName="istio-proxy" Oct 30 15:42:16.004714 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.004677 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" containerName="revoke-keys" Oct 30 15:42:16.007766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.007746 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.029392 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.029368 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:42:16.029527 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.029479 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:42:16.030126 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.030106 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:42:16.030286 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.030269 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:42:16.032181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.032156 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr"] Oct 30 15:42:16.072752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.072723 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.072882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.072782 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ztw\" (UniqueName: \"kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.072946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.072915 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.173715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.173678 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.173886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.173742 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ztw\" (UniqueName: \"kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.173886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.173768 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.188872 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.188845 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:42:16.189015 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.188986 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:42:16.194750 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.194697 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.196343 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.196314 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.211275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.211249 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:42:16.219209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.219185 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ztw\" (UniqueName: \"kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw\") pod \"maas-api-revoke-sub-40327f65-xf9rr\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.335731 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.335713 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:42:16.341082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.341065 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:16.637886 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:16.637812 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr"] Oct 30 15:42:16.640289 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:42:16.640262 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a661e93_7487_42ad_ae6b_394b4e37b459.slice/crio-77571190dafc5577f59df5c489f836f956914b2b63e2cda54e108120939a6350 WatchSource:0}: Error finding container 77571190dafc5577f59df5c489f836f956914b2b63e2cda54e108120939a6350: Status 404 returned error can't find the container with id 77571190dafc5577f59df5c489f836f956914b2b63e2cda54e108120939a6350 Oct 30 15:42:17.402926 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:17.402896 6554 generic.go:356] "Generic (PLEG): container finished" podID="8a661e93-7487-42ad-ae6b-394b4e37b459" containerID="0c9d86798073c2e32b0cd14f8233e76f201fc437a88fb56a04014246d1d3b362" exitCode=0 Oct 30 15:42:17.403269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:17.402953 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" event={"ID":"8a661e93-7487-42ad-ae6b-394b4e37b459","Type":"ContainerDied","Data":"0c9d86798073c2e32b0cd14f8233e76f201fc437a88fb56a04014246d1d3b362"} Oct 30 15:42:17.403269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:17.403013 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" event={"ID":"8a661e93-7487-42ad-ae6b-394b4e37b459","Type":"ContainerStarted","Data":"77571190dafc5577f59df5c489f836f956914b2b63e2cda54e108120939a6350"} Oct 30 15:42:18.538820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.538796 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:18.597043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.597018 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca\") pod \"8a661e93-7487-42ad-ae6b-394b4e37b459\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " Oct 30 15:42:18.597194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.597070 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw\" (UniqueName: \"kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw\") pod \"8a661e93-7487-42ad-ae6b-394b4e37b459\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " Oct 30 15:42:18.597194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.597173 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca\") pod \"8a661e93-7487-42ad-ae6b-394b4e37b459\" (UID: \"8a661e93-7487-42ad-ae6b-394b4e37b459\") " Oct 30 15:42:18.597623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.597562 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca" pod "8a661e93-7487-42ad-ae6b-394b4e37b459" (UID: "8a661e93-7487-42ad-ae6b-394b4e37b459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:18.599237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.599217 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca" pod "8a661e93-7487-42ad-ae6b-394b4e37b459" (UID: "8a661e93-7487-42ad-ae6b-394b4e37b459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:42:18.599338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.599319 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw" pod "8a661e93-7487-42ad-ae6b-394b4e37b459" (UID: "8a661e93-7487-42ad-ae6b-394b4e37b459"). InnerVolumeSpecName "kube-api-access-76ztw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:18.698084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.698059 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:18.698084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.698081 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a661e93-7487-42ad-ae6b-394b4e37b459-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:18.698220 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:18.698090 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ztw\" (UniqueName: \"kubernetes.io/projected/8a661e93-7487-42ad-ae6b-394b4e37b459-kube-api-access-76ztw\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:19.414901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:19.414868 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" Oct 30 15:42:19.415059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:19.414904 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr" event={"ID":"8a661e93-7487-42ad-ae6b-394b4e37b459","Type":"ContainerDied","Data":"77571190dafc5577f59df5c489f836f956914b2b63e2cda54e108120939a6350"} Oct 30 15:42:19.415059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:19.414952 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77571190dafc5577f59df5c489f836f956914b2b63e2cda54e108120939a6350" Oct 30 15:42:19.897526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:19.897493 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:19.897941 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:19.897565 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:20.939460 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:20.939399 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerName="payload-processing" containerID="cri-o://110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f" gracePeriod=25 Oct 30 15:42:20.975563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:20.975361 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" podUID="87357b67-8af1-423f-b92c-5370575b46e1" containerName="payload-pre-processing" containerID="cri-o://12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd" gracePeriod=25 Oct 30 15:42:21.084725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.084700 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:42:21.102673 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.102653 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:42:21.117842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.117814 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume\") pod \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " Oct 30 15:42:21.117950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.117852 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27\" (UniqueName: \"kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27\") pod \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\" (UID: \"7eda1b3c-8310-46b5-a4dd-e89bcc886869\") " Oct 30 15:42:21.118190 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.118169 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume" pod "7eda1b3c-8310-46b5-a4dd-e89bcc886869" (UID: "7eda1b3c-8310-46b5-a4dd-e89bcc886869"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:21.119890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.119870 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27" pod "7eda1b3c-8310-46b5-a4dd-e89bcc886869" (UID: "7eda1b3c-8310-46b5-a4dd-e89bcc886869"). InnerVolumeSpecName "kube-api-access-2ss27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:21.218471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.218439 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr\" (UniqueName: \"kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr\") pod \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " Oct 30 15:42:21.218471 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.218472 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume\") pod \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\" (UID: \"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940\") " Oct 30 15:42:21.218680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.218670 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eda1b3c-8310-46b5-a4dd-e89bcc886869-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:21.218747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.218681 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ss27\" (UniqueName: \"kubernetes.io/projected/7eda1b3c-8310-46b5-a4dd-e89bcc886869-kube-api-access-2ss27\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:21.218874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.218854 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume" pod "1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" (UID: "1cbc1fbb-13b4-4151-b7e0-23cf57b0e940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:21.220502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.220481 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr" pod "1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" (UID: "1cbc1fbb-13b4-4151-b7e0-23cf57b0e940"). InnerVolumeSpecName "kube-api-access-q46tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:21.320076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.320023 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q46tr\" (UniqueName: \"kubernetes.io/projected/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-kube-api-access-q46tr\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:21.320076 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.320045 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:21.425462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.425433 6554 generic.go:356] "Generic (PLEG): container finished" podID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerID="20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557" exitCode=0 Oct 30 15:42:21.425591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.425486 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" Oct 30 15:42:21.425591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.425517 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" event={"ID":"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940","Type":"ContainerDied","Data":"20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557"} Oct 30 15:42:21.425591 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.425575 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9" event={"ID":"1cbc1fbb-13b4-4151-b7e0-23cf57b0e940","Type":"ContainerDied","Data":"29a0d67b99624a15c11e80a0b29fa0ec1645001e63758d6bdf3785b702458ab3"} Oct 30 15:42:21.425783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.425616 6554 scope.go:122] "RemoveContainer" containerID="20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557" Oct 30 15:42:21.427078 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.427052 6554 generic.go:356] "Generic (PLEG): container finished" podID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerID="379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0" exitCode=0 Oct 30 15:42:21.427195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.427108 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" Oct 30 15:42:21.427195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.427106 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" event={"ID":"7eda1b3c-8310-46b5-a4dd-e89bcc886869","Type":"ContainerDied","Data":"379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0"} Oct 30 15:42:21.427324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.427214 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc" event={"ID":"7eda1b3c-8310-46b5-a4dd-e89bcc886869","Type":"ContainerDied","Data":"4883fd352743078498ef96234eaf7228e5168409b8a23e16d615ced15107e669"} Oct 30 15:42:21.445587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.445567 6554 scope.go:122] "RemoveContainer" containerID="20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557" Oct 30 15:42:21.445894 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:21.445872 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557\": container with ID starting with 20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557 not found: ID does not exist" containerID="20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557" Oct 30 15:42:21.445953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.445904 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557"} err="failed to get container status \"20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557\": rpc error: code = NotFound desc = could not find container \"20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557\": container with ID starting with 20b1952cf2fce9250eb58c00ebf3de71bdda57813f16a4be682281cbff758557 not found: ID does not exist" Oct 30 15:42:21.445953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.445923 6554 scope.go:122] "RemoveContainer" containerID="379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0" Oct 30 15:42:21.454236 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.454217 6554 scope.go:122] "RemoveContainer" containerID="379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0" Oct 30 15:42:21.454532 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:21.454514 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0\": container with ID starting with 379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0 not found: ID does not exist" containerID="379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0" Oct 30 15:42:21.454607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.454537 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0"} err="failed to get container status \"379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0\": rpc error: code = NotFound desc = could not find container \"379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0\": container with ID starting with 379d9e6330bab8be97542c56f92097de16a6b216dfad20ea455ebc49e6bb5dd0 not found: ID does not exist" Oct 30 15:42:21.498545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.498525 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc"] Oct 30 15:42:21.502310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.502290 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-a1654b-7bc6db7bc4-knzxc"] Oct 30 15:42:21.540441 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.540418 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9"] Oct 30 15:42:21.544245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:21.544223 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-a1654b-7fbc8cd5ff-bxjc9"] Oct 30 15:42:22.153029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:22.152999 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" path="/var/lib/kubelet/pods/1cbc1fbb-13b4-4151-b7e0-23cf57b0e940/volumes" Oct 30 15:42:22.153432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:22.153416 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" path="/var/lib/kubelet/pods/7eda1b3c-8310-46b5-a4dd-e89bcc886869/volumes" Oct 30 15:42:25.327081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.327040 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" containerID="cri-o://1b2ed7a3e3c577a9570d6db784a8c8faab63e2313e59cf7ccd2365e1e5aeaf41" gracePeriod=45 Oct 30 15:42:25.448861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.448833 6554 generic.go:356] "Generic (PLEG): container finished" podID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerID="1b2ed7a3e3c577a9570d6db784a8c8faab63e2313e59cf7ccd2365e1e5aeaf41" exitCode=0 Oct 30 15:42:25.448997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.448910 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" event={"ID":"c01767e6-485f-4aa7-bcde-9ab19983e054","Type":"ContainerDied","Data":"1b2ed7a3e3c577a9570d6db784a8c8faab63e2313e59cf7ccd2365e1e5aeaf41"} Oct 30 15:42:25.571774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.571749 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:42:25.650805 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.650782 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home\") pod \"c01767e6-485f-4aa7-bcde-9ab19983e054\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " Oct 30 15:42:25.650951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.650814 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs\" (UniqueName: \"kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs\") pod \"c01767e6-485f-4aa7-bcde-9ab19983e054\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " Oct 30 15:42:25.650951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.650851 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir\") pod \"c01767e6-485f-4aa7-bcde-9ab19983e054\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " Oct 30 15:42:25.650951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.650881 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm\") pod \"c01767e6-485f-4aa7-bcde-9ab19983e054\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " Oct 30 15:42:25.650951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.650902 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache\") pod \"c01767e6-485f-4aa7-bcde-9ab19983e054\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " Oct 30 15:42:25.651202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651002 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl\" (UniqueName: \"kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl\") pod \"c01767e6-485f-4aa7-bcde-9ab19983e054\" (UID: \"c01767e6-485f-4aa7-bcde-9ab19983e054\") " Oct 30 15:42:25.651202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651053 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home" pod "c01767e6-485f-4aa7-bcde-9ab19983e054" (UID: "c01767e6-485f-4aa7-bcde-9ab19983e054"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.651202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651095 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir" pod "c01767e6-485f-4aa7-bcde-9ab19983e054" (UID: "c01767e6-485f-4aa7-bcde-9ab19983e054"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.651395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651270 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache" pod "c01767e6-485f-4aa7-bcde-9ab19983e054" (UID: "c01767e6-485f-4aa7-bcde-9ab19983e054"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.651395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651369 6554 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-home\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.651395 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651390 6554 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-tmp-dir\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.651571 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.651404 6554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-model-cache\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.653186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.653159 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs" pod "c01767e6-485f-4aa7-bcde-9ab19983e054" (UID: "c01767e6-485f-4aa7-bcde-9ab19983e054"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:42:25.653306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.653237 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl" pod "c01767e6-485f-4aa7-bcde-9ab19983e054" (UID: "c01767e6-485f-4aa7-bcde-9ab19983e054"). InnerVolumeSpecName "kube-api-access-2xwtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:25.653306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.653294 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm" pod "c01767e6-485f-4aa7-bcde-9ab19983e054" (UID: "c01767e6-485f-4aa7-bcde-9ab19983e054"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 15:42:25.752677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.752645 6554 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c01767e6-485f-4aa7-bcde-9ab19983e054-dshm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.752677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.752675 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwtl\" (UniqueName: \"kubernetes.io/projected/c01767e6-485f-4aa7-bcde-9ab19983e054-kube-api-access-2xwtl\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:25.752867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:25.752688 6554 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c01767e6-485f-4aa7-bcde-9ab19983e054-tls-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:26.454635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:26.454560 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" Oct 30 15:42:26.454635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:26.454602 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg" event={"ID":"c01767e6-485f-4aa7-bcde-9ab19983e054","Type":"ContainerDied","Data":"31aab81ad715f8884517412eb6fe8621a381ed2621309553d4b2ad98d66585eb"} Oct 30 15:42:26.455122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:26.454650 6554 scope.go:122] "RemoveContainer" containerID="1b2ed7a3e3c577a9570d6db784a8c8faab63e2313e59cf7ccd2365e1e5aeaf41" Oct 30 15:42:26.509574 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:26.509548 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg"] Oct 30 15:42:26.523434 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:26.523413 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-83ff4e/distinct-w1-83ff4e-kserve-5cdb458d94-7srzg"] Oct 30 15:42:28.151818 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:28.151788 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" path="/var/lib/kubelet/pods/c01767e6-485f-4aa7-bcde-9ab19983e054/volumes" Oct 30 15:42:30.242667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:30.242630 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" podUID="87357b67-8af1-423f-b92c-5370575b46e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:30.243052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:30.242634 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:31.896045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:31.896013 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv"] Oct 30 15:42:31.910865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:31.910838 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-83ff4e-rdpbv"] Oct 30 15:42:32.152689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:32.152610 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a" path="/var/lib/kubelet/pods/3e0b2ba7-e03e-4a29-868d-cf8361cb3c2a/volumes" Oct 30 15:42:39.202544 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:39.202508 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:42:40.242897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:40.242861 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" podUID="87357b67-8af1-423f-b92c-5370575b46e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:40.242897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:40.242877 6554 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 15:42:41.217783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.217750 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 15:42:41.247393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.247370 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 15:42:41.281967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.281942 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp\" (UniqueName: \"kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp\") pod \"d69746aa-ea97-4e2e-ae58-22716e1c8160\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " Oct 30 15:42:41.282098 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.281999 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume\") pod \"d69746aa-ea97-4e2e-ae58-22716e1c8160\" (UID: \"d69746aa-ea97-4e2e-ae58-22716e1c8160\") " Oct 30 15:42:41.282708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.282438 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume" pod "d69746aa-ea97-4e2e-ae58-22716e1c8160" (UID: "d69746aa-ea97-4e2e-ae58-22716e1c8160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:41.284799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.284780 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp" pod "d69746aa-ea97-4e2e-ae58-22716e1c8160" (UID: "d69746aa-ea97-4e2e-ae58-22716e1c8160"). InnerVolumeSpecName "kube-api-access-974zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:41.383567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.383545 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm\" (UniqueName: \"kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm\") pod \"87357b67-8af1-423f-b92c-5370575b46e1\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " Oct 30 15:42:41.383694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.383674 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume\") pod \"87357b67-8af1-423f-b92c-5370575b46e1\" (UID: \"87357b67-8af1-423f-b92c-5370575b46e1\") " Oct 30 15:42:41.383840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.383826 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d69746aa-ea97-4e2e-ae58-22716e1c8160-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:41.383918 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.383841 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974zp\" (UniqueName: \"kubernetes.io/projected/d69746aa-ea97-4e2e-ae58-22716e1c8160-kube-api-access-974zp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:41.384019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.383997 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume" pod "87357b67-8af1-423f-b92c-5370575b46e1" (UID: "87357b67-8af1-423f-b92c-5370575b46e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:42:41.385481 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.385461 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm" pod "87357b67-8af1-423f-b92c-5370575b46e1" (UID: "87357b67-8af1-423f-b92c-5370575b46e1"). InnerVolumeSpecName "kube-api-access-c5rpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:42:41.484597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.484544 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5rpm\" (UniqueName: \"kubernetes.io/projected/87357b67-8af1-423f-b92c-5370575b46e1-kube-api-access-c5rpm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:41.484597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.484564 6554 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87357b67-8af1-423f-b92c-5370575b46e1-plugins-config-volume\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:42:41.529739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.529713 6554 generic.go:356] "Generic (PLEG): container finished" podID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerID="110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f" exitCode=0 Oct 30 15:42:41.529837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.529760 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" Oct 30 15:42:41.529837 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.529793 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" event={"ID":"d69746aa-ea97-4e2e-ae58-22716e1c8160","Type":"ContainerDied","Data":"110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f"} Oct 30 15:42:41.529967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.529843 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts" event={"ID":"d69746aa-ea97-4e2e-ae58-22716e1c8160","Type":"ContainerDied","Data":"0d19036a499f2103e877346ccd7179c8808f1a51e20fb09a7d0e87e44dd83313"} Oct 30 15:42:41.529967 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.529865 6554 scope.go:122] "RemoveContainer" containerID="110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f" Oct 30 15:42:41.531244 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.531215 6554 generic.go:356] "Generic (PLEG): container finished" podID="87357b67-8af1-423f-b92c-5370575b46e1" containerID="12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd" exitCode=0 Oct 30 15:42:41.531363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.531250 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" Oct 30 15:42:41.531363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.531284 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" event={"ID":"87357b67-8af1-423f-b92c-5370575b46e1","Type":"ContainerDied","Data":"12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd"} Oct 30 15:42:41.531363 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.531322 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x" event={"ID":"87357b67-8af1-423f-b92c-5370575b46e1","Type":"ContainerDied","Data":"3d5a67e8f8058505b1a936dc295060ad20267afb3100be1159d69c7448c8f2cc"} Oct 30 15:42:41.539615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.539599 6554 scope.go:122] "RemoveContainer" containerID="110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f" Oct 30 15:42:41.539877 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:41.539857 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f\": container with ID starting with 110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f not found: ID does not exist" containerID="110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f" Oct 30 15:42:41.539960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.539886 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f"} err="failed to get container status \"110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f\": rpc error: code = NotFound desc = could not find container \"110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f\": container with ID starting with 110ecc56c044c6c338c98bee01c2e91b165bc9941763c5a1c7bb0a4e4cdb832f not found: ID does not exist" Oct 30 15:42:41.539960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.539912 6554 scope.go:122] "RemoveContainer" containerID="12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd" Oct 30 15:42:41.547762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.547744 6554 scope.go:122] "RemoveContainer" containerID="12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd" Oct 30 15:42:41.548015 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:42:41.547987 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd\": container with ID starting with 12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd not found: ID does not exist" containerID="12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd" Oct 30 15:42:41.548066 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.548015 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd"} err="failed to get container status \"12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd\": rpc error: code = NotFound desc = could not find container \"12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd\": container with ID starting with 12cfc421d48ca2a379515d7e1c9ead31453c5004ceae923a2b10543a715fe5cd not found: ID does not exist" Oct 30 15:42:41.604257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.604239 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x"] Oct 30 15:42:41.610293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.610266 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-83ff4e-bb546f56c-zh52x"] Oct 30 15:42:41.656030 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.656007 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts"] Oct 30 15:42:41.685698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:41.685678 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-83ff4e-7ff95979dd-tswts"] Oct 30 15:42:42.153050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:42.153023 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87357b67-8af1-423f-b92c-5370575b46e1" path="/var/lib/kubelet/pods/87357b67-8af1-423f-b92c-5370575b46e1/volumes" Oct 30 15:42:42.153412 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:42.153397 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" path="/var/lib/kubelet/pods/d69746aa-ea97-4e2e-ae58-22716e1c8160/volumes" Oct 30 15:42:58.534975 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.534939 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm"] Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535295 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535307 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535320 6554 cpu_manager.go:414] "containerMap: removing container" podUID="87357b67-8af1-423f-b92c-5370575b46e1" containerName="payload-pre-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535326 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87357b67-8af1-423f-b92c-5370575b46e1" containerName="payload-pre-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535363 6554 cpu_manager.go:414] "containerMap: removing container" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerName="payload-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535368 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerName="payload-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535376 6554 cpu_manager.go:414] "containerMap: removing container" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerName="payload-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535380 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerName="payload-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535387 6554 cpu_manager.go:414] "containerMap: removing container" podUID="8a661e93-7487-42ad-ae6b-394b4e37b459" containerName="revoke-keys" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535392 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a661e93-7487-42ad-ae6b-394b4e37b459" containerName="revoke-keys" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535403 6554 cpu_manager.go:414] "containerMap: removing container" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerName="payload-pre-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535408 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerName="payload-pre-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535465 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="d69746aa-ea97-4e2e-ae58-22716e1c8160" containerName="payload-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535475 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a661e93-7487-42ad-ae6b-394b4e37b459" containerName="revoke-keys" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535482 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cbc1fbb-13b4-4151-b7e0-23cf57b0e940" containerName="payload-pre-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535491 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c01767e6-485f-4aa7-bcde-9ab19983e054" containerName="main" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535497 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="87357b67-8af1-423f-b92c-5370575b46e1" containerName="payload-pre-processing" Oct 30 15:42:58.537766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.535507 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eda1b3c-8310-46b5-a4dd-e89bcc886869" containerName="payload-processing" Oct 30 15:42:58.538708 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.538691 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.557809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.557782 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-khr2k\"" Oct 30 15:42:58.586862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.586834 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm"] Oct 30 15:42:58.626068 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626041 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626076 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626185 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626100 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626216 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626268 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626257 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2759d\" (UniqueName: \"kubernetes.io/projected/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-kube-api-access-2759d\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626323 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626370 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626392 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.626477 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.626446 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.727683 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727656 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2759d\" (UniqueName: \"kubernetes.io/projected/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-kube-api-access-2759d\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.727817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727689 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.727817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727710 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.727817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727726 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.727817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727748 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.727817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727809 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727830 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727863 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728092 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.727894 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.728292 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.728332 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728505 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.728413 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.728674 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.728831 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.728681 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.730203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.730180 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.730666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.730645 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.755752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.755728 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2759d\" (UniqueName: \"kubernetes.io/projected/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-kube-api-access-2759d\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.759129 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.759107 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-9drkm\" (UID: \"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:58.851475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:58.851425 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:42:59.165562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.165529 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm"] Oct 30 15:42:59.167616 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:42:59.167588 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1a9bc0_b6f3_48c5_888e_0d44cd70e74f.slice/crio-68403f7bffb644351b31be1fc9043e2eab7fa50f8a9c8f0ecc8ca3cfce09f60d WatchSource:0}: Error finding container 68403f7bffb644351b31be1fc9043e2eab7fa50f8a9c8f0ecc8ca3cfce09f60d: Status 404 returned error can't find the container with id 68403f7bffb644351b31be1fc9043e2eab7fa50f8a9c8f0ecc8ca3cfce09f60d Oct 30 15:42:59.169765 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.169733 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:42:59.169878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.169821 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:42:59.169878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.169851 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:42:59.619896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.619812 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" event={"ID":"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f","Type":"ContainerStarted","Data":"1f1e74603a261f982ddc48a1862d9080a42a24a3300053be6c061d98cc2a1046"} Oct 30 15:42:59.619896 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.619864 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" event={"ID":"7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f","Type":"ContainerStarted","Data":"68403f7bffb644351b31be1fc9043e2eab7fa50f8a9c8f0ecc8ca3cfce09f60d"} Oct 30 15:42:59.683488 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.683422 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podStartSLOduration=1.683403006 podStartE2EDuration="1.683403006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:42:59.682123791 +0000 UTC m=+3283.641165197" watchObservedRunningTime="2026-10-30 15:42:59.683403006 +0000 UTC m=+3283.642444427" Oct 30 15:42:59.852486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:42:59.852447 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:43:00.852964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:00.852929 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9drkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 15:43:00.853465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:00.852984 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podUID="7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 15:43:01.852426 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:01.852394 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9drkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 15:43:01.852607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:01.852444 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podUID="7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 15:43:02.852282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:02.852245 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9drkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 15:43:02.852704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:02.852301 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podUID="7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 15:43:03.851993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:03.851953 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9drkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 15:43:03.852205 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:03.852006 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podUID="7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 15:43:04.852159 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:04.852120 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9drkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 15:43:04.852606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:04.852171 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podUID="7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 15:43:05.852113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:05.852075 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-9drkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 15:43:05.852295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:05.852149 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" podUID="7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 15:43:07.159009 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:07.158976 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:43:07.159537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:07.159279 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:43:07.160388 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:07.160365 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-9drkm" Oct 30 15:43:09.191095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.191062 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84"] Oct 30 15:43:09.195119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.195091 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.209704 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:43:09.209675 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:43:09.212302 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.212281 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:43:09.226243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.226222 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:43:09.226414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.226310 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:43:09.226496 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.226410 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:43:09.227784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.227765 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84"] Oct 30 15:43:09.238018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.237992 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.238139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.238050 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wqsq\" (UniqueName: \"kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.238139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.238080 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.338435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.338416 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.338533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.338448 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wqsq\" (UniqueName: \"kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.338533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.338468 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.339010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.338988 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.340845 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.340828 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.363424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.363402 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wqsq\" (UniqueName: \"kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq\") pod \"maas-api-revoke-sub-85237e45-f5x84\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.514455 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.514396 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:09.811083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:09.811060 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84"] Oct 30 15:43:09.812909 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:43:09.812881 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc849e1f0_28fa_465f_a4f0_434d6e607c67.slice/crio-4de336ece1132d1e2858b3749d7bd02efe217f4f59208cbf066391fc0196fb21 WatchSource:0}: Error finding container 4de336ece1132d1e2858b3749d7bd02efe217f4f59208cbf066391fc0196fb21: Status 404 returned error can't find the container with id 4de336ece1132d1e2858b3749d7bd02efe217f4f59208cbf066391fc0196fb21 Oct 30 15:43:10.670880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:10.670854 6554 generic.go:356] "Generic (PLEG): container finished" podID="c849e1f0-28fa-465f-a4f0-434d6e607c67" containerID="b854493c3d2d8c502fa2cc10ca3785528d4a3e9d498819395d1a947894b131ce" exitCode=0 Oct 30 15:43:10.671182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:10.670940 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" event={"ID":"c849e1f0-28fa-465f-a4f0-434d6e607c67","Type":"ContainerDied","Data":"b854493c3d2d8c502fa2cc10ca3785528d4a3e9d498819395d1a947894b131ce"} Oct 30 15:43:10.671182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:10.670998 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" event={"ID":"c849e1f0-28fa-465f-a4f0-434d6e607c67","Type":"ContainerStarted","Data":"4de336ece1132d1e2858b3749d7bd02efe217f4f59208cbf066391fc0196fb21"} Oct 30 15:43:11.809786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.809765 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:11.859039 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.859011 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca\") pod \"c849e1f0-28fa-465f-a4f0-434d6e607c67\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " Oct 30 15:43:11.859179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.859086 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca\") pod \"c849e1f0-28fa-465f-a4f0-434d6e607c67\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " Oct 30 15:43:11.859179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.859147 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq\" (UniqueName: \"kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq\") pod \"c849e1f0-28fa-465f-a4f0-434d6e607c67\" (UID: \"c849e1f0-28fa-465f-a4f0-434d6e607c67\") " Oct 30 15:43:11.859699 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.859559 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca" pod "c849e1f0-28fa-465f-a4f0-434d6e607c67" (UID: "c849e1f0-28fa-465f-a4f0-434d6e607c67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:43:11.861947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.861925 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca" pod "c849e1f0-28fa-465f-a4f0-434d6e607c67" (UID: "c849e1f0-28fa-465f-a4f0-434d6e607c67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:43:11.862292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.862269 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq" pod "c849e1f0-28fa-465f-a4f0-434d6e607c67" (UID: "c849e1f0-28fa-465f-a4f0-434d6e607c67"). InnerVolumeSpecName "kube-api-access-9wqsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:43:11.960686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.960594 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wqsq\" (UniqueName: \"kubernetes.io/projected/c849e1f0-28fa-465f-a4f0-434d6e607c67-kube-api-access-9wqsq\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:43:11.960686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.960631 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:43:11.960686 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:11.960650 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c849e1f0-28fa-465f-a4f0-434d6e607c67-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:43:12.683652 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:12.683618 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" Oct 30 15:43:12.683861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:12.683624 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84" event={"ID":"c849e1f0-28fa-465f-a4f0-434d6e607c67","Type":"ContainerDied","Data":"4de336ece1132d1e2858b3749d7bd02efe217f4f59208cbf066391fc0196fb21"} Oct 30 15:43:12.683861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:43:12.683746 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4de336ece1132d1e2858b3749d7bd02efe217f4f59208cbf066391fc0196fb21" Oct 30 15:43:39.200630 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:43:39.200590 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:44:01.438084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.437994 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v"] Oct 30 15:44:01.438624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.438362 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c849e1f0-28fa-465f-a4f0-434d6e607c67" containerName="revoke-keys" Oct 30 15:44:01.438624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.438374 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c849e1f0-28fa-465f-a4f0-434d6e607c67" containerName="revoke-keys" Oct 30 15:44:01.438624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.438463 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c849e1f0-28fa-465f-a4f0-434d6e607c67" containerName="revoke-keys" Oct 30 15:44:01.441925 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.441906 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.461364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.461278 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:44:01.461794 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.461523 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:44:01.475119 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.475082 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:44:01.475243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.475210 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:44:01.476637 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.476615 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v"] Oct 30 15:44:01.485691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.485664 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.485828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.485695 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.485828 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.485788 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqrm\" (UniqueName: \"kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.586368 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.586328 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.586500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.586386 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.586500 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.586436 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqrm\" (UniqueName: \"kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.587113 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.587096 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.588709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.588686 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.616074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.616052 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqrm\" (UniqueName: \"kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm\") pod \"maas-api-revoke-sub-c1e271a9-8kj4v\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:01.755393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:01.755327 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:02.057625 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:02.057599 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v"] Oct 30 15:44:02.060129 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:44:02.060100 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21e3d76_0ab9_46b3_9716_0b8d4a0f24f1.slice/crio-0e4ab20185a7281e4053e8c321541190e4581972acf8b22ece342490d0ecb31e WatchSource:0}: Error finding container 0e4ab20185a7281e4053e8c321541190e4581972acf8b22ece342490d0ecb31e: Status 404 returned error can't find the container with id 0e4ab20185a7281e4053e8c321541190e4581972acf8b22ece342490d0ecb31e Oct 30 15:44:02.922752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:02.922720 6554 generic.go:356] "Generic (PLEG): container finished" podID="d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" containerID="27af8cda9f74651b0167b6a3b08aab7c5c0b464e83b8f5449a2745addc7041ae" exitCode=0 Oct 30 15:44:02.923072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:02.922794 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" event={"ID":"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1","Type":"ContainerDied","Data":"27af8cda9f74651b0167b6a3b08aab7c5c0b464e83b8f5449a2745addc7041ae"} Oct 30 15:44:02.923072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:02.922872 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" event={"ID":"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1","Type":"ContainerStarted","Data":"0e4ab20185a7281e4053e8c321541190e4581972acf8b22ece342490d0ecb31e"} Oct 30 15:44:04.054502 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.054477 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:04.112186 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.112162 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm\" (UniqueName: \"kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm\") pod \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " Oct 30 15:44:04.112309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.112250 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca\") pod \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " Oct 30 15:44:04.112373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.112327 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca\") pod \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\" (UID: \"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1\") " Oct 30 15:44:04.112674 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.112644 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca" pod "d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" (UID: "d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:44:04.114272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.114246 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm" pod "d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" (UID: "d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1"). InnerVolumeSpecName "kube-api-access-wxqrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:44:04.114393 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.114317 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca" pod "d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" (UID: "d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:44:04.213321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.213266 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:04.213321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.213290 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxqrm\" (UniqueName: \"kubernetes.io/projected/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-kube-api-access-wxqrm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:04.213321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.213301 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:04.932823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.932795 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" Oct 30 15:44:04.932991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.932800 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v" event={"ID":"d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1","Type":"ContainerDied","Data":"0e4ab20185a7281e4053e8c321541190e4581972acf8b22ece342490d0ecb31e"} Oct 30 15:44:04.932991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:04.932942 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e4ab20185a7281e4053e8c321541190e4581972acf8b22ece342490d0ecb31e" Oct 30 15:44:09.200680 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:44:09.200645 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:44:13.775454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.775424 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h"] Oct 30 15:44:13.776373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.776328 6554 cpu_manager.go:414] "containerMap: removing container" podUID="d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" containerName="revoke-keys" Oct 30 15:44:13.776373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.776365 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" containerName="revoke-keys" Oct 30 15:44:13.776524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.776432 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" containerName="revoke-keys" Oct 30 15:44:13.779751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.779730 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.805282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.805258 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h"] Oct 30 15:44:13.814136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.814112 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm"] Oct 30 15:44:13.817539 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.817522 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.839835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.839814 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm"] Oct 30 15:44:13.894301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894276 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.894480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894326 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894429 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894464 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faca140-9640-4e89-9dc2-77479c5e0f64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894493 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2775a872-b309-41d5-a34c-cf822339e896-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.894627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894523 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894588 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894627 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894617 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2775a872-b309-41d5-a34c-cf822339e896-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.894833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894639 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.894833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894763 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894833 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894797 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894835 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7524\" (UniqueName: \"kubernetes.io/projected/5faca140-9640-4e89-9dc2-77479c5e0f64-kube-api-access-b7524\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.894961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894868 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.894961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894918 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.894961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894945 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5hnb\" (UniqueName: \"kubernetes.io/projected/2775a872-b309-41d5-a34c-cf822339e896-kube-api-access-d5hnb\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.895105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.894969 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.895105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.895007 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.895105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.895032 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2775a872-b309-41d5-a34c-cf822339e896-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996220 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996249 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996266 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faca140-9640-4e89-9dc2-77479c5e0f64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996283 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2775a872-b309-41d5-a34c-cf822339e896-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996410 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996448 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996474 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2775a872-b309-41d5-a34c-cf822339e896-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996506 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996582 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996566 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996598 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996639 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7524\" (UniqueName: \"kubernetes.io/projected/5faca140-9640-4e89-9dc2-77479c5e0f64-kube-api-access-b7524\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996667 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996700 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996706 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996735 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5hnb\" (UniqueName: \"kubernetes.io/projected/2775a872-b309-41d5-a34c-cf822339e896-kube-api-access-d5hnb\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996774 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996808 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.996867 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996833 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2775a872-b309-41d5-a34c-cf822339e896-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.997416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996898 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.997416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.996945 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.997416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997174 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2775a872-b309-41d5-a34c-cf822339e896-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.997416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997177 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.997416 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997223 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.997732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997496 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.997732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997512 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.997732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997519 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.997899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997743 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5faca140-9640-4e89-9dc2-77479c5e0f64-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.997899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.997776 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.999081 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.999054 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2775a872-b309-41d5-a34c-cf822339e896-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.999278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.999256 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2775a872-b309-41d5-a34c-cf822339e896-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:13.999383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.999271 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:13.999383 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:13.999290 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:14.022248 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.022223 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5faca140-9640-4e89-9dc2-77479c5e0f64-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:14.023169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.023140 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2775a872-b309-41d5-a34c-cf822339e896-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:14.023491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.023472 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7524\" (UniqueName: \"kubernetes.io/projected/5faca140-9640-4e89-9dc2-77479c5e0f64-kube-api-access-b7524\") pod \"maas-default-gateway-openshift-default-6cc458d887-jdv8h\" (UID: \"5faca140-9640-4e89-9dc2-77479c5e0f64\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:14.023598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.023579 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5hnb\" (UniqueName: \"kubernetes.io/projected/2775a872-b309-41d5-a34c-cf822339e896-kube-api-access-d5hnb\") pod \"maas-default-gateway-openshift-default-6cc458d887-twvbm\" (UID: \"2775a872-b309-41d5-a34c-cf822339e896\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:14.093682 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.093630 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:14.131930 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.131902 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:14.410510 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.410483 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h"] Oct 30 15:44:14.412636 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:44:14.412607 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5faca140_9640_4e89_9dc2_77479c5e0f64.slice/crio-ee37f166763eefcef2167e1a7aad714351c1ad063dc7d4a700d827fc2ca8c2e8 WatchSource:0}: Error finding container ee37f166763eefcef2167e1a7aad714351c1ad063dc7d4a700d827fc2ca8c2e8: Status 404 returned error can't find the container with id ee37f166763eefcef2167e1a7aad714351c1ad063dc7d4a700d827fc2ca8c2e8 Oct 30 15:44:14.414589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.414558 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:44:14.414666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.414623 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:44:14.414666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.414652 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:44:14.456866 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.456843 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm"] Oct 30 15:44:14.457461 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:44:14.457434 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2775a872_b309_41d5_a34c_cf822339e896.slice/crio-0591c626b28a04bd8274c016c032efbe3ab004ef271c2f9dcb376bc3a07fd3b0 WatchSource:0}: Error finding container 0591c626b28a04bd8274c016c032efbe3ab004ef271c2f9dcb376bc3a07fd3b0: Status 404 returned error can't find the container with id 0591c626b28a04bd8274c016c032efbe3ab004ef271c2f9dcb376bc3a07fd3b0 Oct 30 15:44:14.459620 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.459585 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:44:14.459698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.459662 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:44:14.459739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.459704 6554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 15:44:15.000070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.999066 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" event={"ID":"5faca140-9640-4e89-9dc2-77479c5e0f64","Type":"ContainerStarted","Data":"fa8218d7f5a5f637ba7391e935acb1fcff25d3ee181f7f083c8f6ea0eccf2029"} Oct 30 15:44:15.000070 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:14.999136 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" event={"ID":"5faca140-9640-4e89-9dc2-77479c5e0f64","Type":"ContainerStarted","Data":"ee37f166763eefcef2167e1a7aad714351c1ad063dc7d4a700d827fc2ca8c2e8"} Oct 30 15:44:15.009806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.008733 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" event={"ID":"2775a872-b309-41d5-a34c-cf822339e896","Type":"ContainerStarted","Data":"7887c7edf706ca542f3160f3e71bb616ff92dc9a64dece179a453a666845ada5"} Oct 30 15:44:15.009806 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.008816 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" event={"ID":"2775a872-b309-41d5-a34c-cf822339e896","Type":"ContainerStarted","Data":"0591c626b28a04bd8274c016c032efbe3ab004ef271c2f9dcb376bc3a07fd3b0"} Oct 30 15:44:15.096902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.095413 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:15.098290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.098244 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:15.098804 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.098522 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:15.132829 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.132811 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:15.134370 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.134334 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:15.134474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.134390 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:15.138919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.138875 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podStartSLOduration=2.13886291 podStartE2EDuration="2.13886291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:15.137141776 +0000 UTC m=+3359.096183183" watchObservedRunningTime="2026-10-30 15:44:15.13886291 +0000 UTC m=+3359.097904315" Oct 30 15:44:15.139588 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:15.139561 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podStartSLOduration=2.139544632 podStartE2EDuration="2.139544632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:15.066012667 +0000 UTC m=+3359.025054073" watchObservedRunningTime="2026-10-30 15:44:15.139544632 +0000 UTC m=+3359.098586037" Oct 30 15:44:16.095751 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:16.095712 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:16.096399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:16.096365 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:16.132951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:16.132922 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:16.133067 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:16.132964 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:17.110480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:17.110440 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:17.111013 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:17.110514 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:17.132470 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:17.132438 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:17.132689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:17.132523 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:18.094747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:18.094705 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:18.095042 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:18.095010 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:18.132924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:18.132894 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:18.133278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:18.132940 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:19.097376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:19.096714 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:19.097376 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:19.096777 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:19.132942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:19.132907 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:19.133621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:19.133573 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:20.094716 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:20.094675 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:20.095118 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:20.095078 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:20.133008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:20.132980 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:20.133401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:20.133022 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:21.094050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:21.094003 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" start-of-body= Oct 30 15:44:21.094257 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:21.094068 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": dial tcp 10.134.0.242:15021: connect: connection refused" Oct 30 15:44:21.132929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:21.132889 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body= Oct 30 15:44:21.133082 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:21.132939 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" Oct 30 15:44:23.095196 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:23.095146 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:23.095764 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:23.095224 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:23.132429 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:23.132399 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:23.132646 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:23.132612 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:24.137492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.137454 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:24.137938 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.137526 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:24.175583 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.175510 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-twvbm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:24.175773 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.175580 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" podUID="2775a872-b309-41d5-a34c-cf822339e896" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:24.901266 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.901234 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm"] Oct 30 15:44:24.906389 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.906362 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:24.907497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.906795 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:24.907497 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.907249 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:24.919550 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.919525 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm"] Oct 30 15:44:24.931095 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.931074 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:44:24.931265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.931078 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:44:24.931371 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.931090 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:44:24.937927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:24.937906 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:44:25.028400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.028342 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.028600 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.028504 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.028680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.028610 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bb26\" (UniqueName: \"kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.129467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.129430 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.129633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.129521 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.129633 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.129594 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bb26\" (UniqueName: \"kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.130149 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.130125 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.132107 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.132081 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.156285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.156213 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bb26\" (UniqueName: \"kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26\") pod \"maas-api-revoke-sub-2e1f5b5f-m6cbm\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.178494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.178465 6554 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-jdv8h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 15:44:25.178624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.178519 6554 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" podUID="5faca140-9640-4e89-9dc2-77479c5e0f64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 15:44:25.186180 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.186100 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-twvbm" Oct 30 15:44:25.223689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.223667 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:25.368344 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.368304 6554 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:25.368657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.368632 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:25.373264 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.373242 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-jdv8h" Oct 30 15:44:25.541309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:25.541286 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm"] Oct 30 15:44:25.543801 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:44:25.543777 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b76ef6_7342_4597_b034_a35a453f791d.slice/crio-5c7974a8d5de6f3a033499f0a6c1eab526c089ed3ebda270388803ad0ea56f04 WatchSource:0}: Error finding container 5c7974a8d5de6f3a033499f0a6c1eab526c089ed3ebda270388803ad0ea56f04: Status 404 returned error can't find the container with id 5c7974a8d5de6f3a033499f0a6c1eab526c089ed3ebda270388803ad0ea56f04 Oct 30 15:44:26.075281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:26.075240 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" event={"ID":"c0b76ef6-7342-4597-b034-a35a453f791d","Type":"ContainerStarted","Data":"e196bf6d0d193fceafdd59d991a323f51d46557c45351dd083df90e2df6293cf"} Oct 30 15:44:26.075447 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:26.075306 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" event={"ID":"c0b76ef6-7342-4597-b034-a35a453f791d","Type":"ContainerStarted","Data":"5c7974a8d5de6f3a033499f0a6c1eab526c089ed3ebda270388803ad0ea56f04"} Oct 30 15:44:26.120569 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:26.120496 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" podStartSLOduration=2.120484375 podStartE2EDuration="2.120484375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:44:26.119166123 +0000 UTC m=+3370.078207527" watchObservedRunningTime="2026-10-30 15:44:26.120484375 +0000 UTC m=+3370.079525779" Oct 30 15:44:27.081853 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:27.081815 6554 generic.go:356] "Generic (PLEG): container finished" podID="c0b76ef6-7342-4597-b034-a35a453f791d" containerID="e196bf6d0d193fceafdd59d991a323f51d46557c45351dd083df90e2df6293cf" exitCode=0 Oct 30 15:44:27.082243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:27.081898 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" event={"ID":"c0b76ef6-7342-4597-b034-a35a453f791d","Type":"ContainerDied","Data":"e196bf6d0d193fceafdd59d991a323f51d46557c45351dd083df90e2df6293cf"} Oct 30 15:44:28.217254 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.217231 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:28.358823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.358757 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca\") pod \"c0b76ef6-7342-4597-b034-a35a453f791d\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " Oct 30 15:44:28.358823 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.358795 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca\") pod \"c0b76ef6-7342-4597-b034-a35a453f791d\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " Oct 30 15:44:28.358991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.358842 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26\" (UniqueName: \"kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26\") pod \"c0b76ef6-7342-4597-b034-a35a453f791d\" (UID: \"c0b76ef6-7342-4597-b034-a35a453f791d\") " Oct 30 15:44:28.359106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.359084 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca" pod "c0b76ef6-7342-4597-b034-a35a453f791d" (UID: "c0b76ef6-7342-4597-b034-a35a453f791d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:44:28.359275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.359251 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:28.360891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.360863 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca" pod "c0b76ef6-7342-4597-b034-a35a453f791d" (UID: "c0b76ef6-7342-4597-b034-a35a453f791d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:44:28.360988 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.360949 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26" pod "c0b76ef6-7342-4597-b034-a35a453f791d" (UID: "c0b76ef6-7342-4597-b034-a35a453f791d"). InnerVolumeSpecName "kube-api-access-7bb26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:44:28.460357 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.460332 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bb26\" (UniqueName: \"kubernetes.io/projected/c0b76ef6-7342-4597-b034-a35a453f791d-kube-api-access-7bb26\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:28.460465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:28.460373 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b76ef6-7342-4597-b034-a35a453f791d-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:29.092524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:29.092489 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" event={"ID":"c0b76ef6-7342-4597-b034-a35a453f791d","Type":"ContainerDied","Data":"5c7974a8d5de6f3a033499f0a6c1eab526c089ed3ebda270388803ad0ea56f04"} Oct 30 15:44:29.092524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:29.092521 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm" Oct 30 15:44:29.092742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:29.092546 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c7974a8d5de6f3a033499f0a6c1eab526c089ed3ebda270388803ad0ea56f04" Oct 30 15:44:36.351874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.351834 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx"] Oct 30 15:44:36.352435 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.352412 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c0b76ef6-7342-4597-b034-a35a453f791d" containerName="revoke-keys" Oct 30 15:44:36.352526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.352436 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b76ef6-7342-4597-b034-a35a453f791d" containerName="revoke-keys" Oct 30 15:44:36.352593 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.352572 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b76ef6-7342-4597-b034-a35a453f791d" containerName="revoke-keys" Oct 30 15:44:36.355959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.355939 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.373004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.372984 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:44:36.374856 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.374831 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx"] Oct 30 15:44:36.386649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.386611 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:44:36.386815 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.386677 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:44:36.386875 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.386817 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:44:36.433733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.433698 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.433854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.433782 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnzws\" (UniqueName: \"kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.433854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.433817 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.534537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.534507 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnzws\" (UniqueName: \"kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.534647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.534556 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.534647 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.534638 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.535301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.535283 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.537064 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.537043 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.559301 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.559282 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnzws\" (UniqueName: \"kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws\") pod \"maas-api-revoke-sub-ec843018-55bbx\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.668188 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.668163 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:36.982962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:36.982922 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx"] Oct 30 15:44:36.984549 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:44:36.984519 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc2fbc3_8a08_4d54_9b2f_3b4427d4cb4c.slice/crio-cb54c9144e0c197929fc100788a77056295be67fbba436014b29ab85ad760674 WatchSource:0}: Error finding container cb54c9144e0c197929fc100788a77056295be67fbba436014b29ab85ad760674: Status 404 returned error can't find the container with id cb54c9144e0c197929fc100788a77056295be67fbba436014b29ab85ad760674 Oct 30 15:44:37.132664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:37.132633 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" event={"ID":"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c","Type":"ContainerStarted","Data":"cb54c9144e0c197929fc100788a77056295be67fbba436014b29ab85ad760674"} Oct 30 15:44:38.138971 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:38.138937 6554 generic.go:356] "Generic (PLEG): container finished" podID="9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" containerID="5b51cf6c1a5e07fb053a307e4ec86d31b3fcdc2b9b959fee033c67aaf09fa910" exitCode=0 Oct 30 15:44:38.139479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:38.139025 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" event={"ID":"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c","Type":"ContainerDied","Data":"5b51cf6c1a5e07fb053a307e4ec86d31b3fcdc2b9b959fee033c67aaf09fa910"} Oct 30 15:44:39.206502 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:44:39.206466 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:44:39.295937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.295915 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:39.466202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.466126 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca\") pod \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " Oct 30 15:44:39.466332 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.466210 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws\" (UniqueName: \"kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws\") pod \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " Oct 30 15:44:39.466414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.466333 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca\") pod \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\" (UID: \"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c\") " Oct 30 15:44:39.466508 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.466485 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca" pod "9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" (UID: "9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:44:39.466676 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.466653 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:39.468239 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.468218 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws" pod "9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" (UID: "9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c"). InnerVolumeSpecName "kube-api-access-vnzws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:44:39.468362 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.468316 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca" pod "9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" (UID: "9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:44:39.568072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.568046 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnzws\" (UniqueName: \"kubernetes.io/projected/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-kube-api-access-vnzws\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:39.568072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:39.568071 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:44:40.153387 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:40.153341 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" Oct 30 15:44:40.156121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:40.156087 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx" event={"ID":"9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c","Type":"ContainerDied","Data":"cb54c9144e0c197929fc100788a77056295be67fbba436014b29ab85ad760674"} Oct 30 15:44:40.156281 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:44:40.156144 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb54c9144e0c197929fc100788a77056295be67fbba436014b29ab85ad760674" Oct 30 15:45:00.144369 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.144328 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv"] Oct 30 15:45:00.144850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.144783 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" containerName="revoke-keys" Oct 30 15:45:00.144850 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.144802 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" containerName="revoke-keys" Oct 30 15:45:00.144937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.144924 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" containerName="revoke-keys" Oct 30 15:45:00.148408 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.148389 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.169747 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.169716 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv"] Oct 30 15:45:00.180913 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.180889 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:45:00.181086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.180922 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:45:00.181086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.181036 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:45:00.181342 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.181321 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8ftk\" (UniqueName: \"kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk\") pod \"maas-api-key-cleanup-29889585-cmplv\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.181428 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.181412 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889585-cmplv\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.282774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.282739 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8ftk\" (UniqueName: \"kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk\") pod \"maas-api-key-cleanup-29889585-cmplv\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.282961 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.282818 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889585-cmplv\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.283454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.283434 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889585-cmplv\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.307626 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.307599 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8ftk\" (UniqueName: \"kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk\") pod \"maas-api-key-cleanup-29889585-cmplv\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.463912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.463858 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:00.764245 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:00.764223 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv"] Oct 30 15:45:00.766323 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:45:00.766291 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ea8ce0_1bf6_42f9_8dd6_8d3d75d26c91.slice/crio-25d8c0220f2596f0bac6a41bb00400c76fa7d94acaf9f4b7fcabda049af547ce WatchSource:0}: Error finding container 25d8c0220f2596f0bac6a41bb00400c76fa7d94acaf9f4b7fcabda049af547ce: Status 404 returned error can't find the container with id 25d8c0220f2596f0bac6a41bb00400c76fa7d94acaf9f4b7fcabda049af547ce Oct 30 15:45:01.266296 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:01.266262 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" event={"ID":"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91","Type":"ContainerStarted","Data":"ea3ac79da24174a0708d98426a62aca528a9429b6253c1e3fd2de93bebe63f39"} Oct 30 15:45:01.266679 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:01.266321 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" event={"ID":"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91","Type":"ContainerStarted","Data":"25d8c0220f2596f0bac6a41bb00400c76fa7d94acaf9f4b7fcabda049af547ce"} Oct 30 15:45:01.311409 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:01.311319 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" podStartSLOduration=1.311306978 podStartE2EDuration="1.311306978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:45:01.309614384 +0000 UTC m=+3405.268655789" watchObservedRunningTime="2026-10-30 15:45:01.311306978 +0000 UTC m=+3405.270348430" Oct 30 15:45:02.272182 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:02.272143 6554 generic.go:356] "Generic (PLEG): container finished" podID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerID="ea3ac79da24174a0708d98426a62aca528a9429b6253c1e3fd2de93bebe63f39" exitCode=60 Oct 30 15:45:02.272598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:02.272225 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" event={"ID":"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91","Type":"ContainerDied","Data":"ea3ac79da24174a0708d98426a62aca528a9429b6253c1e3fd2de93bebe63f39"} Oct 30 15:45:02.272598 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:02.272436 6554 scope.go:122] "RemoveContainer" containerID="ea3ac79da24174a0708d98426a62aca528a9429b6253c1e3fd2de93bebe63f39" Oct 30 15:45:03.285179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:03.285103 6554 generic.go:356] "Generic (PLEG): container finished" podID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerID="bdc2274a57e5e09b0bfd056b8151d0fdc021b8ed8426b86d1f3869dbafec7c98" exitCode=60 Oct 30 15:45:03.285639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:03.285227 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" event={"ID":"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91","Type":"ContainerDied","Data":"bdc2274a57e5e09b0bfd056b8151d0fdc021b8ed8426b86d1f3869dbafec7c98"} Oct 30 15:45:03.285639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:03.285307 6554 scope.go:122] "RemoveContainer" containerID="ea3ac79da24174a0708d98426a62aca528a9429b6253c1e3fd2de93bebe63f39" Oct 30 15:45:03.285639 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:03.285398 6554 scope.go:122] "RemoveContainer" containerID="bdc2274a57e5e09b0bfd056b8151d0fdc021b8ed8426b86d1f3869dbafec7c98" Oct 30 15:45:03.285819 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:45:03.285661 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889585-cmplv_odh-ai-gateway-infra(66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" Oct 30 15:45:04.291079 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:04.291053 6554 scope.go:122] "RemoveContainer" containerID="bdc2274a57e5e09b0bfd056b8151d0fdc021b8ed8426b86d1f3869dbafec7c98" Oct 30 15:45:04.291511 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:45:04.291264 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889585-cmplv_odh-ai-gateway-infra(66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" Oct 30 15:45:09.200450 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:45:09.200417 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:45:15.147482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:15.147453 6554 scope.go:122] "RemoveContainer" containerID="bdc2274a57e5e09b0bfd056b8151d0fdc021b8ed8426b86d1f3869dbafec7c98" Oct 30 15:45:16.358404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:16.358362 6554 generic.go:356] "Generic (PLEG): container finished" podID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerID="10a002eae7caafb7dd144aa10b89a433dd8941db55278f735646ce199655d4d5" exitCode=60 Oct 30 15:45:16.358404 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:16.358383 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" event={"ID":"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91","Type":"ContainerDied","Data":"10a002eae7caafb7dd144aa10b89a433dd8941db55278f735646ce199655d4d5"} Oct 30 15:45:16.358923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:16.358463 6554 scope.go:122] "RemoveContainer" containerID="bdc2274a57e5e09b0bfd056b8151d0fdc021b8ed8426b86d1f3869dbafec7c98" Oct 30 15:45:16.358923 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:16.358763 6554 scope.go:122] "RemoveContainer" containerID="10a002eae7caafb7dd144aa10b89a433dd8941db55278f735646ce199655d4d5" Oct 30 15:45:16.359036 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:45:16.359017 6554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889585-cmplv_odh-ai-gateway-infra(66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" Oct 30 15:45:17.411097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.411063 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv"] Oct 30 15:45:17.567739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.567712 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:17.636069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.636031 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk\" (UniqueName: \"kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk\") pod \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " Oct 30 15:45:17.636258 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.636140 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca\") pod \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\" (UID: \"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91\") " Oct 30 15:45:17.636556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.636526 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca" pod "66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" (UID: "66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:45:17.636700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.636638 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-maas-api-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:17.638783 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.638753 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk" pod "66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" (UID: "66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91"). InnerVolumeSpecName "kube-api-access-f8ftk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:17.737832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:17.737759 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8ftk\" (UniqueName: \"kubernetes.io/projected/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91-kube-api-access-f8ftk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:18.370581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:18.370546 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" event={"ID":"66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91","Type":"ContainerDied","Data":"25d8c0220f2596f0bac6a41bb00400c76fa7d94acaf9f4b7fcabda049af547ce"} Oct 30 15:45:18.370581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:18.370567 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv" Oct 30 15:45:18.370808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:18.370627 6554 scope.go:122] "RemoveContainer" containerID="10a002eae7caafb7dd144aa10b89a433dd8941db55278f735646ce199655d4d5" Oct 30 15:45:18.424931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:18.424879 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv"] Oct 30 15:45:18.438555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:18.438531 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889585-cmplv"] Oct 30 15:45:20.154088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154050 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" path="/var/lib/kubelet/pods/66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91/volumes" Oct 30 15:45:20.154513 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154378 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r"] Oct 30 15:45:20.154740 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154725 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.154779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154741 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.154779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154754 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.154779 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154759 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.154885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154850 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.154885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154858 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.154885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.154864 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:20.158028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.158012 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.178018 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.177987 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r"] Oct 30 15:45:20.189774 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.189750 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:45:20.189907 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.189865 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:45:20.189991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.189934 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:45:20.190057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.189989 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:45:20.261566 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.261541 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.261685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.261595 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgr6j\" (UniqueName: \"kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.261733 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.261690 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.362857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.362833 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgr6j\" (UniqueName: \"kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.362980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.362876 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.362980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.362946 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.363669 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.363646 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.365260 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.365239 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.387140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.387115 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgr6j\" (UniqueName: \"kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j\") pod \"maas-api-revoke-sub-d3fba056-kzh4r\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.470899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.470850 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:20.767865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:20.767842 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r"] Oct 30 15:45:20.769415 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:45:20.769387 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca5fe0ef_94d2_47e9_80a0_4fbbd6ae6d82.slice/crio-d7e58ba6102e77f609364d47a26847e55f2013a8aa036a9ba140833f572ce4be WatchSource:0}: Error finding container d7e58ba6102e77f609364d47a26847e55f2013a8aa036a9ba140833f572ce4be: Status 404 returned error can't find the container with id d7e58ba6102e77f609364d47a26847e55f2013a8aa036a9ba140833f572ce4be Oct 30 15:45:21.386049 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:21.386017 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" event={"ID":"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82","Type":"ContainerStarted","Data":"547a416f963a1e512ae9d9d753eb46ab01f851d59066839cae75c0e01990a53a"} Oct 30 15:45:21.386543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:21.386080 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" event={"ID":"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82","Type":"ContainerStarted","Data":"d7e58ba6102e77f609364d47a26847e55f2013a8aa036a9ba140833f572ce4be"} Oct 30 15:45:21.437964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:21.437894 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" podStartSLOduration=1.437874358 podStartE2EDuration="1.437874358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:45:21.436459761 +0000 UTC m=+3425.395501179" watchObservedRunningTime="2026-10-30 15:45:21.437874358 +0000 UTC m=+3425.396915765" Oct 30 15:45:22.407218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:22.407181 6554 generic.go:356] "Generic (PLEG): container finished" podID="ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" containerID="547a416f963a1e512ae9d9d753eb46ab01f851d59066839cae75c0e01990a53a" exitCode=0 Oct 30 15:45:22.407887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:22.407860 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" event={"ID":"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82","Type":"ContainerDied","Data":"547a416f963a1e512ae9d9d753eb46ab01f851d59066839cae75c0e01990a53a"} Oct 30 15:45:23.898457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:23.895762 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:24.036732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.036646 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j\" (UniqueName: \"kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j\") pod \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " Oct 30 15:45:24.036991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.036964 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca\") pod \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " Oct 30 15:45:24.037106 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.037032 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca\") pod \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\" (UID: \"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82\") " Oct 30 15:45:24.037494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.037463 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca" pod "ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" (UID: "ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:45:24.039653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.039627 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca" pod "ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" (UID: "ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:45:24.039767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.039738 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j" pod "ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" (UID: "ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82"). InnerVolumeSpecName "kube-api-access-cgr6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:24.138069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.138030 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:24.138069 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.138070 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgr6j\" (UniqueName: \"kubernetes.io/projected/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-kube-api-access-cgr6j\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:24.138242 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.138085 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:24.435207 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.435179 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" event={"ID":"ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82","Type":"ContainerDied","Data":"d7e58ba6102e77f609364d47a26847e55f2013a8aa036a9ba140833f572ce4be"} Oct 30 15:45:24.435378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.435224 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7e58ba6102e77f609364d47a26847e55f2013a8aa036a9ba140833f572ce4be" Oct 30 15:45:24.435378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:24.435189 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r" Oct 30 15:45:39.200881 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:45:39.200837 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:45:41.879882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.879844 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh"] Oct 30 15:45:41.880380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.880200 6554 cpu_manager.go:414] "containerMap: removing container" podUID="ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" containerName="revoke-keys" Oct 30 15:45:41.880380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.880213 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" containerName="revoke-keys" Oct 30 15:45:41.880380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.880244 6554 cpu_manager.go:414] "containerMap: removing container" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:41.880380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.880249 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ea8ce0-1bf6-42f9-8dd6-8d3d75d26c91" containerName="cleanup" Oct 30 15:45:41.880380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.880316 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" containerName="revoke-keys" Oct 30 15:45:41.883320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.883298 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:41.903907 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.903875 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh"] Oct 30 15:45:41.915490 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.915461 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:45:41.915624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.915520 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:45:41.915624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.915562 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:45:41.915624 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.915587 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:45:41.949832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.949798 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:41.949946 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.949931 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28p6c\" (UniqueName: \"kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:41.949989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:41.949961 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.050552 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.050528 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28p6c\" (UniqueName: \"kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.050680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.050561 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.050680 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.050633 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.051320 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.051301 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.052992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.052969 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.075962 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.075935 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28p6c\" (UniqueName: \"kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c\") pod \"maas-api-revoke-sub-ae594979-bksbh\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.198729 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.198707 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:42.496752 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.496727 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh"] Oct 30 15:45:42.498718 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:45:42.498689 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50317320_f011_4db5_98d9_5836b6dc601b.slice/crio-85b5be66d89e1ada16b1cf69ffd2bc287eb4e0866f1586784af3bcdd55b24a7b WatchSource:0}: Error finding container 85b5be66d89e1ada16b1cf69ffd2bc287eb4e0866f1586784af3bcdd55b24a7b: Status 404 returned error can't find the container with id 85b5be66d89e1ada16b1cf69ffd2bc287eb4e0866f1586784af3bcdd55b24a7b Oct 30 15:45:42.550688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:42.550660 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" event={"ID":"50317320-f011-4db5-98d9-5836b6dc601b","Type":"ContainerStarted","Data":"85b5be66d89e1ada16b1cf69ffd2bc287eb4e0866f1586784af3bcdd55b24a7b"} Oct 30 15:45:43.557515 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:43.557446 6554 generic.go:356] "Generic (PLEG): container finished" podID="50317320-f011-4db5-98d9-5836b6dc601b" containerID="2d44cd523bb5f735e9475252ea7313cb6a1f44d8333fb792a2bb16a14ce8f176" exitCode=0 Oct 30 15:45:43.557960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:43.557531 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" event={"ID":"50317320-f011-4db5-98d9-5836b6dc601b","Type":"ContainerDied","Data":"2d44cd523bb5f735e9475252ea7313cb6a1f44d8333fb792a2bb16a14ce8f176"} Oct 30 15:45:44.693659 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.693632 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:44.771993 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.771962 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c\" (UniqueName: \"kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c\") pod \"50317320-f011-4db5-98d9-5836b6dc601b\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " Oct 30 15:45:44.772134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.772008 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca\") pod \"50317320-f011-4db5-98d9-5836b6dc601b\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " Oct 30 15:45:44.772134 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.772062 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca\") pod \"50317320-f011-4db5-98d9-5836b6dc601b\" (UID: \"50317320-f011-4db5-98d9-5836b6dc601b\") " Oct 30 15:45:44.772543 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.772514 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca" pod "50317320-f011-4db5-98d9-5836b6dc601b" (UID: "50317320-f011-4db5-98d9-5836b6dc601b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:45:44.774209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.774185 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c" pod "50317320-f011-4db5-98d9-5836b6dc601b" (UID: "50317320-f011-4db5-98d9-5836b6dc601b"). InnerVolumeSpecName "kube-api-access-28p6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:45:44.774209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.774185 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca" pod "50317320-f011-4db5-98d9-5836b6dc601b" (UID: "50317320-f011-4db5-98d9-5836b6dc601b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:45:44.873088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.873030 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28p6c\" (UniqueName: \"kubernetes.io/projected/50317320-f011-4db5-98d9-5836b6dc601b-kube-api-access-28p6c\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:44.873088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.873058 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:44.873088 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:44.873068 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50317320-f011-4db5-98d9-5836b6dc601b-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:45:45.569040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:45.569002 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" Oct 30 15:45:45.569040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:45.569032 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh" event={"ID":"50317320-f011-4db5-98d9-5836b6dc601b","Type":"ContainerDied","Data":"85b5be66d89e1ada16b1cf69ffd2bc287eb4e0866f1586784af3bcdd55b24a7b"} Oct 30 15:45:45.569312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:45:45.569079 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85b5be66d89e1ada16b1cf69ffd2bc287eb4e0866f1586784af3bcdd55b24a7b" Oct 30 15:46:03.446761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.446730 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4"] Oct 30 15:46:03.447237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.447171 6554 cpu_manager.go:414] "containerMap: removing container" podUID="50317320-f011-4db5-98d9-5836b6dc601b" containerName="revoke-keys" Oct 30 15:46:03.447237 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.447184 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50317320-f011-4db5-98d9-5836b6dc601b" containerName="revoke-keys" Oct 30 15:46:03.447315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.447250 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="50317320-f011-4db5-98d9-5836b6dc601b" containerName="revoke-keys" Oct 30 15:46:03.450621 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.450604 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.481844 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.481818 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:46:03.481980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.481876 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:46:03.481980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.481913 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:46:03.481980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.481973 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:46:03.488372 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.485175 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4"] Oct 30 15:46:03.576534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.576504 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v2jk\" (UniqueName: \"kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.576651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.576556 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.576698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.576660 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.677209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.677185 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v2jk\" (UniqueName: \"kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.677340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.677234 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.677340 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.677298 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.677977 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.677956 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.679578 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.679560 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.702037 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.701978 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v2jk\" (UniqueName: \"kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk\") pod \"maas-api-revoke-sub-a4aef7d7-hfcf4\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:03.763709 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:03.763689 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:04.065762 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:04.065723 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4"] Oct 30 15:46:04.067407 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:46:04.067373 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8fee90_3c09_4e5c_88b0_55a1f65cfa09.slice/crio-ae07a789e73498b8146955ebffbd543eaf46fa43a9c1ee787d9bbd4941307e2b WatchSource:0}: Error finding container ae07a789e73498b8146955ebffbd543eaf46fa43a9c1ee787d9bbd4941307e2b: Status 404 returned error can't find the container with id ae07a789e73498b8146955ebffbd543eaf46fa43a9c1ee787d9bbd4941307e2b Oct 30 15:46:04.667519 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:04.667487 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" event={"ID":"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09","Type":"ContainerStarted","Data":"50495df48b230e2dbc7de2dccd993f880e2071e62ecf2fc66294a78b22946d78"} Oct 30 15:46:04.667910 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:04.667537 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" event={"ID":"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09","Type":"ContainerStarted","Data":"ae07a789e73498b8146955ebffbd543eaf46fa43a9c1ee787d9bbd4941307e2b"} Oct 30 15:46:04.712619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:04.712575 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" podStartSLOduration=1.7125616689999998 podStartE2EDuration="1.712561669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:46:04.710442204 +0000 UTC m=+3468.669483611" watchObservedRunningTime="2026-10-30 15:46:04.712561669 +0000 UTC m=+3468.671603096" Oct 30 15:46:05.674165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:05.674131 6554 generic.go:356] "Generic (PLEG): container finished" podID="9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" containerID="50495df48b230e2dbc7de2dccd993f880e2071e62ecf2fc66294a78b22946d78" exitCode=0 Oct 30 15:46:05.674563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:05.674221 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" event={"ID":"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09","Type":"ContainerDied","Data":"50495df48b230e2dbc7de2dccd993f880e2071e62ecf2fc66294a78b22946d78"} Oct 30 15:46:06.807622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.807599 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:06.904458 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.904431 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk\" (UniqueName: \"kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk\") pod \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " Oct 30 15:46:06.904613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.904489 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca\") pod \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " Oct 30 15:46:06.904613 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.904511 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca\") pod \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\" (UID: \"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09\") " Oct 30 15:46:06.904908 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.904883 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca" pod "9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" (UID: "9d8fee90-3c09-4e5c-88b0-55a1f65cfa09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:46:06.906501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.906477 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca" pod "9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" (UID: "9d8fee90-3c09-4e5c-88b0-55a1f65cfa09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:46:06.906616 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:06.906511 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk" pod "9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" (UID: "9d8fee90-3c09-4e5c-88b0-55a1f65cfa09"). InnerVolumeSpecName "kube-api-access-6v2jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:07.005802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:07.005753 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v2jk\" (UniqueName: \"kubernetes.io/projected/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-kube-api-access-6v2jk\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:07.005802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:07.005774 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:07.005802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:07.005783 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:07.685137 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:07.685105 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" Oct 30 15:46:07.685298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:07.685137 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4" event={"ID":"9d8fee90-3c09-4e5c-88b0-55a1f65cfa09","Type":"ContainerDied","Data":"ae07a789e73498b8146955ebffbd543eaf46fa43a9c1ee787d9bbd4941307e2b"} Oct 30 15:46:07.685298 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:07.685191 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae07a789e73498b8146955ebffbd543eaf46fa43a9c1ee787d9bbd4941307e2b" Oct 30 15:46:09.200765 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:46:09.200729 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:46:23.963287 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.963255 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q"] Oct 30 15:46:23.963796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.963705 6554 cpu_manager.go:414] "containerMap: removing container" podUID="9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" containerName="revoke-keys" Oct 30 15:46:23.963796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.963719 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" containerName="revoke-keys" Oct 30 15:46:23.963796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.963784 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" containerName="revoke-keys" Oct 30 15:46:23.966951 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.966928 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:23.984589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.984494 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:46:23.998075 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.998053 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:46:23.998252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.998115 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:46:23.998252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:23.998145 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:46:24.000063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.000037 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q"] Oct 30 15:46:24.052755 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.052729 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8tkm\" (UniqueName: \"kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.052889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.052770 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.052889 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.052881 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.154272 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.154244 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.154432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.154334 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tkm\" (UniqueName: \"kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.154432 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.154382 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.154964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.154932 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.156615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.156589 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.179540 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.179508 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tkm\" (UniqueName: \"kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm\") pod \"maas-api-revoke-sub-79805ca9-vx26q\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.279295 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.279232 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:24.581001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.580978 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q"] Oct 30 15:46:24.582579 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:46:24.582542 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b97c9a_f5c1_445b_9e7b_2b3df688db57.slice/crio-4dad60a79cb189fadf36a6ac00ef5a34250718987191411baa8f26d5ba8c09c4 WatchSource:0}: Error finding container 4dad60a79cb189fadf36a6ac00ef5a34250718987191411baa8f26d5ba8c09c4: Status 404 returned error can't find the container with id 4dad60a79cb189fadf36a6ac00ef5a34250718987191411baa8f26d5ba8c09c4 Oct 30 15:46:24.769746 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:24.769717 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" event={"ID":"69b97c9a-f5c1-445b-9e7b-2b3df688db57","Type":"ContainerStarted","Data":"4dad60a79cb189fadf36a6ac00ef5a34250718987191411baa8f26d5ba8c09c4"} Oct 30 15:46:25.781166 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:25.781133 6554 generic.go:356] "Generic (PLEG): container finished" podID="69b97c9a-f5c1-445b-9e7b-2b3df688db57" containerID="4dda96161844de6bfeaa8d2eca5db2902905ff98cb991f906ac90031a287f926" exitCode=0 Oct 30 15:46:25.781657 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:25.781226 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" event={"ID":"69b97c9a-f5c1-445b-9e7b-2b3df688db57","Type":"ContainerDied","Data":"4dda96161844de6bfeaa8d2eca5db2902905ff98cb991f906ac90031a287f926"} Oct 30 15:46:26.919491 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:26.919463 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:27.082987 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.082927 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm\" (UniqueName: \"kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm\") pod \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " Oct 30 15:46:27.083110 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.083025 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca\") pod \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " Oct 30 15:46:27.083158 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.083123 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca\") pod \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\" (UID: \"69b97c9a-f5c1-445b-9e7b-2b3df688db57\") " Oct 30 15:46:27.083485 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.083460 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca" pod "69b97c9a-f5c1-445b-9e7b-2b3df688db57" (UID: "69b97c9a-f5c1-445b-9e7b-2b3df688db57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:46:27.085053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.085028 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm" pod "69b97c9a-f5c1-445b-9e7b-2b3df688db57" (UID: "69b97c9a-f5c1-445b-9e7b-2b3df688db57"). InnerVolumeSpecName "kube-api-access-c8tkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:27.085053 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.085045 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca" pod "69b97c9a-f5c1-445b-9e7b-2b3df688db57" (UID: "69b97c9a-f5c1-445b-9e7b-2b3df688db57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:46:27.184330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.184306 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:27.184330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.184327 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69b97c9a-f5c1-445b-9e7b-2b3df688db57-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:27.184492 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.184336 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8tkm\" (UniqueName: \"kubernetes.io/projected/69b97c9a-f5c1-445b-9e7b-2b3df688db57-kube-api-access-c8tkm\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:27.791811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.791784 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" Oct 30 15:46:27.791997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.791809 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q" event={"ID":"69b97c9a-f5c1-445b-9e7b-2b3df688db57","Type":"ContainerDied","Data":"4dad60a79cb189fadf36a6ac00ef5a34250718987191411baa8f26d5ba8c09c4"} Oct 30 15:46:27.791997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:27.791861 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dad60a79cb189fadf36a6ac00ef5a34250718987191411baa8f26d5ba8c09c4" Oct 30 15:46:29.027504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:29.027469 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g"] Oct 30 15:46:29.041575 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:29.041545 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb5cb392-g8h6g"] Oct 30 15:46:30.152250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:30.152205 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0407377-1192-4591-9515-1f6b9c40a9c2" path="/var/lib/kubelet/pods/c0407377-1192-4591-9515-1f6b9c40a9c2/volumes" Oct 30 15:46:39.246496 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:46:39.246460 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:46:44.005139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.005106 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd"] Oct 30 15:46:44.005700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.005530 6554 cpu_manager.go:414] "containerMap: removing container" podUID="69b97c9a-f5c1-445b-9e7b-2b3df688db57" containerName="revoke-keys" Oct 30 15:46:44.005700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.005544 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b97c9a-f5c1-445b-9e7b-2b3df688db57" containerName="revoke-keys" Oct 30 15:46:44.005700 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.005651 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b97c9a-f5c1-445b-9e7b-2b3df688db57" containerName="revoke-keys" Oct 30 15:46:44.009874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.009854 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.029028 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.029002 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd"] Oct 30 15:46:44.040694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.040670 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:46:44.040825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.040705 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:46:44.041169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.041146 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:46:44.041169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.041162 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:46:44.065729 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.065706 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.065820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.065735 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqgz5\" (UniqueName: \"kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.065820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.065761 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.166854 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.166833 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.166959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.166869 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqgz5\" (UniqueName: \"kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.166959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.166891 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.167599 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.167581 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.169133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.169116 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.191618 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.191594 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqgz5\" (UniqueName: \"kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5\") pod \"maas-api-revoke-sub-0b7e8edb-dnhnd\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.322258 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.322201 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:44.624901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.624876 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd"] Oct 30 15:46:44.627916 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:46:44.627882 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ff11a0_7083_4251_8660_934e24e9c2d5.slice/crio-d29a2875b1939c929ffecd4a9a4532275b1822016952cb7a4caa489c341f576d WatchSource:0}: Error finding container d29a2875b1939c929ffecd4a9a4532275b1822016952cb7a4caa489c341f576d: Status 404 returned error can't find the container with id d29a2875b1939c929ffecd4a9a4532275b1822016952cb7a4caa489c341f576d Oct 30 15:46:44.876312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:44.876257 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" event={"ID":"a0ff11a0-7083-4251-8660-934e24e9c2d5","Type":"ContainerStarted","Data":"d29a2875b1939c929ffecd4a9a4532275b1822016952cb7a4caa489c341f576d"} Oct 30 15:46:45.882859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:45.882800 6554 generic.go:356] "Generic (PLEG): container finished" podID="a0ff11a0-7083-4251-8660-934e24e9c2d5" containerID="b489b0d83f6eae3cb8cc2602b2dd4be1bb10c2cbb58662cb4b37769a466342c3" exitCode=0 Oct 30 15:46:45.882859 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:45.882853 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" event={"ID":"a0ff11a0-7083-4251-8660-934e24e9c2d5","Type":"ContainerDied","Data":"b489b0d83f6eae3cb8cc2602b2dd4be1bb10c2cbb58662cb4b37769a466342c3"} Oct 30 15:46:47.016160 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.016137 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:47.092282 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.092256 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca\") pod \"a0ff11a0-7083-4251-8660-934e24e9c2d5\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " Oct 30 15:46:47.092418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.092309 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca\") pod \"a0ff11a0-7083-4251-8660-934e24e9c2d5\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " Oct 30 15:46:47.092418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.092380 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5\" (UniqueName: \"kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5\") pod \"a0ff11a0-7083-4251-8660-934e24e9c2d5\" (UID: \"a0ff11a0-7083-4251-8660-934e24e9c2d5\") " Oct 30 15:46:47.092691 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.092663 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca" pod "a0ff11a0-7083-4251-8660-934e24e9c2d5" (UID: "a0ff11a0-7083-4251-8660-934e24e9c2d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:46:47.094453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.094427 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca" pod "a0ff11a0-7083-4251-8660-934e24e9c2d5" (UID: "a0ff11a0-7083-4251-8660-934e24e9c2d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:46:47.094453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.094444 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5" pod "a0ff11a0-7083-4251-8660-934e24e9c2d5" (UID: "a0ff11a0-7083-4251-8660-934e24e9c2d5"). InnerVolumeSpecName "kube-api-access-fqgz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:47.193374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.193278 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:47.193374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.193305 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0ff11a0-7083-4251-8660-934e24e9c2d5-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:47.193374 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.193315 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqgz5\" (UniqueName: \"kubernetes.io/projected/a0ff11a0-7083-4251-8660-934e24e9c2d5-kube-api-access-fqgz5\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:47.896980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.896935 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" event={"ID":"a0ff11a0-7083-4251-8660-934e24e9c2d5","Type":"ContainerDied","Data":"d29a2875b1939c929ffecd4a9a4532275b1822016952cb7a4caa489c341f576d"} Oct 30 15:46:47.897187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.897041 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d29a2875b1939c929ffecd4a9a4532275b1822016952cb7a4caa489c341f576d" Oct 30 15:46:47.897187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:47.897122 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd" Oct 30 15:46:54.783466 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.783434 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn"] Oct 30 15:46:54.784097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.784019 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a0ff11a0-7083-4251-8660-934e24e9c2d5" containerName="revoke-keys" Oct 30 15:46:54.784097 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.784041 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ff11a0-7083-4251-8660-934e24e9c2d5" containerName="revoke-keys" Oct 30 15:46:54.784238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.784152 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ff11a0-7083-4251-8660-934e24e9c2d5" containerName="revoke-keys" Oct 30 15:46:54.787309 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.787287 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.818784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.818756 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:46:54.818784 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.818773 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:46:54.819012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.818895 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:46:54.819012 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.818966 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:46:54.820787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.820757 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn"] Oct 30 15:46:54.861484 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.861460 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.861614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.861598 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.861658 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.861648 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvvmv\" (UniqueName: \"kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.962615 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.962593 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.962741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.962630 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvvmv\" (UniqueName: \"kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.962741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.962666 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.963311 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.963294 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.964970 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.964953 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:54.987856 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:54.987829 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvvmv\" (UniqueName: \"kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv\") pod \"maas-api-revoke-sub-3d4156f1-h54fn\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:55.100283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:55.100233 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:55.399044 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:55.399013 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn"] Oct 30 15:46:55.400999 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:46:55.400969 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c17a02_1ebe_4a45_bfa1_5e8709a92d3b.slice/crio-07b310be31c5f98f423fce335fc275e926b765b0919f67d56333497e2b354a5b WatchSource:0}: Error finding container 07b310be31c5f98f423fce335fc275e926b765b0919f67d56333497e2b354a5b: Status 404 returned error can't find the container with id 07b310be31c5f98f423fce335fc275e926b765b0919f67d56333497e2b354a5b Oct 30 15:46:55.938654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:55.938570 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" event={"ID":"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b","Type":"ContainerStarted","Data":"651753ce1529e6de77a6c76541fd1acda14db5c9b6a6d37df0b56bfbfda3e018"} Oct 30 15:46:55.938654 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:55.938636 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" event={"ID":"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b","Type":"ContainerStarted","Data":"07b310be31c5f98f423fce335fc275e926b765b0919f67d56333497e2b354a5b"} Oct 30 15:46:55.985959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:55.985904 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" podStartSLOduration=1.985891437 podStartE2EDuration="1.985891437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:46:55.983716663 +0000 UTC m=+3519.942758071" watchObservedRunningTime="2026-10-30 15:46:55.985891437 +0000 UTC m=+3519.944932852" Oct 30 15:46:56.944749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:56.944687 6554 generic.go:356] "Generic (PLEG): container finished" podID="23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" containerID="651753ce1529e6de77a6c76541fd1acda14db5c9b6a6d37df0b56bfbfda3e018" exitCode=0 Oct 30 15:46:56.945105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:56.944751 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" event={"ID":"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b","Type":"ContainerDied","Data":"651753ce1529e6de77a6c76541fd1acda14db5c9b6a6d37df0b56bfbfda3e018"} Oct 30 15:46:58.089154 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.089128 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:46:58.189729 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.189689 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv\" (UniqueName: \"kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv\") pod \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " Oct 30 15:46:58.189864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.189762 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca\") pod \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " Oct 30 15:46:58.189864 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.189803 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca\") pod \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\" (UID: \"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b\") " Oct 30 15:46:58.190226 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.190195 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca" pod "23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" (UID: "23c17a02-1ebe-4a45-bfa1-5e8709a92d3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:46:58.191756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.191736 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv" pod "23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" (UID: "23c17a02-1ebe-4a45-bfa1-5e8709a92d3b"). InnerVolumeSpecName "kube-api-access-dvvmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:46:58.191756 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.191741 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca" pod "23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" (UID: "23c17a02-1ebe-4a45-bfa1-5e8709a92d3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:46:58.291526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.291462 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvvmv\" (UniqueName: \"kubernetes.io/projected/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-kube-api-access-dvvmv\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:58.291526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.291494 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:58.291526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.291508 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:46:58.955555 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.955528 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" event={"ID":"23c17a02-1ebe-4a45-bfa1-5e8709a92d3b","Type":"ContainerDied","Data":"07b310be31c5f98f423fce335fc275e926b765b0919f67d56333497e2b354a5b"} Oct 30 15:46:58.955713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.955578 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b310be31c5f98f423fce335fc275e926b765b0919f67d56333497e2b354a5b" Oct 30 15:46:58.955713 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:46:58.955541 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn" Oct 30 15:47:09.247229 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:47:09.247178 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:47:16.204056 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.204022 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn"] Oct 30 15:47:16.217175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.217142 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-h54fn"] Oct 30 15:47:16.255593 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.255564 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9"] Oct 30 15:47:16.256168 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.256149 6554 cpu_manager.go:414] "containerMap: removing container" podUID="23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" containerName="revoke-keys" Oct 30 15:47:16.256221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.256171 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" containerName="revoke-keys" Oct 30 15:47:16.256291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.256279 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" containerName="revoke-keys" Oct 30 15:47:16.260071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.260040 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.275586 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.275561 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.275698 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.275674 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfhrl\" (UniqueName: \"kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.275743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.275730 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.279187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.279149 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:47:16.279514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.279495 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:47:16.279666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.279581 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:47:16.279749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.279730 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9"] Oct 30 15:47:16.290982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.290961 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:47:16.377187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.377165 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.377290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.377259 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfhrl\" (UniqueName: \"kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.377290 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.377287 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.377960 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.377937 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.379474 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.379456 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.416140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.416118 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfhrl\" (UniqueName: \"kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl\") pod \"maas-api-revoke-sub-3d4156f1-58pr9\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.573706 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.573659 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:16.871108 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:16.871086 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9"] Oct 30 15:47:16.872690 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:47:16.872654 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96464ca0_9190_4674_af75_3fe93a97a8c8.slice/crio-7982cc48e02af4f7a205cf7fed26a195521b030e1b00253a8f8cca382fa55404 WatchSource:0}: Error finding container 7982cc48e02af4f7a205cf7fed26a195521b030e1b00253a8f8cca382fa55404: Status 404 returned error can't find the container with id 7982cc48e02af4f7a205cf7fed26a195521b030e1b00253a8f8cca382fa55404 Oct 30 15:47:17.061982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:17.061953 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" event={"ID":"96464ca0-9190-4674-af75-3fe93a97a8c8","Type":"ContainerStarted","Data":"7982cc48e02af4f7a205cf7fed26a195521b030e1b00253a8f8cca382fa55404"} Oct 30 15:47:18.068595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:18.068562 6554 generic.go:356] "Generic (PLEG): container finished" podID="96464ca0-9190-4674-af75-3fe93a97a8c8" containerID="98f8a44d6080706aa9279b63b961bc83587bcf10b18b68ad836d53e0bec15e6a" exitCode=0 Oct 30 15:47:18.069031 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:18.068646 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" event={"ID":"96464ca0-9190-4674-af75-3fe93a97a8c8","Type":"ContainerDied","Data":"98f8a44d6080706aa9279b63b961bc83587bcf10b18b68ad836d53e0bec15e6a"} Oct 30 15:47:18.155840 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:18.154421 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c17a02-1ebe-4a45-bfa1-5e8709a92d3b" path="/var/lib/kubelet/pods/23c17a02-1ebe-4a45-bfa1-5e8709a92d3b/volumes" Oct 30 15:47:19.048631 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.048597 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr"] Oct 30 15:47:19.062443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.062415 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40327f65-xf9rr"] Oct 30 15:47:19.209139 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.209113 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:19.299580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.299518 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl\" (UniqueName: \"kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl\") pod \"96464ca0-9190-4674-af75-3fe93a97a8c8\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " Oct 30 15:47:19.299580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.299556 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca\") pod \"96464ca0-9190-4674-af75-3fe93a97a8c8\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " Oct 30 15:47:19.299790 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.299675 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca\") pod \"96464ca0-9190-4674-af75-3fe93a97a8c8\" (UID: \"96464ca0-9190-4674-af75-3fe93a97a8c8\") " Oct 30 15:47:19.299992 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.299968 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca" pod "96464ca0-9190-4674-af75-3fe93a97a8c8" (UID: "96464ca0-9190-4674-af75-3fe93a97a8c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:47:19.301632 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.301605 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca" pod "96464ca0-9190-4674-af75-3fe93a97a8c8" (UID: "96464ca0-9190-4674-af75-3fe93a97a8c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:47:19.301727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.301684 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl" pod "96464ca0-9190-4674-af75-3fe93a97a8c8" (UID: "96464ca0-9190-4674-af75-3fe93a97a8c8"). InnerVolumeSpecName "kube-api-access-dfhrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:47:19.401397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.401343 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfhrl\" (UniqueName: \"kubernetes.io/projected/96464ca0-9190-4674-af75-3fe93a97a8c8-kube-api-access-dfhrl\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:47:19.401397 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.401396 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:47:19.401534 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:19.401407 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96464ca0-9190-4674-af75-3fe93a97a8c8-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:47:20.082614 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:20.082581 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" Oct 30 15:47:20.082939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:20.082584 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9" event={"ID":"96464ca0-9190-4674-af75-3fe93a97a8c8","Type":"ContainerDied","Data":"7982cc48e02af4f7a205cf7fed26a195521b030e1b00253a8f8cca382fa55404"} Oct 30 15:47:20.082939 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:20.082712 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7982cc48e02af4f7a205cf7fed26a195521b030e1b00253a8f8cca382fa55404" Oct 30 15:47:20.153652 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:20.153613 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a661e93-7487-42ad-ae6b-394b4e37b459" path="/var/lib/kubelet/pods/8a661e93-7487-42ad-ae6b-394b4e37b459/volumes" Oct 30 15:47:39.202564 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:47:39.202524 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:47:40.158754 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.158714 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9"] Oct 30 15:47:40.172310 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.172283 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-58pr9"] Oct 30 15:47:40.206212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.206189 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb"] Oct 30 15:47:40.206563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.206547 6554 cpu_manager.go:414] "containerMap: removing container" podUID="96464ca0-9190-4674-af75-3fe93a97a8c8" containerName="revoke-keys" Oct 30 15:47:40.206563 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.206559 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96464ca0-9190-4674-af75-3fe93a97a8c8" containerName="revoke-keys" Oct 30 15:47:40.206656 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.206646 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="96464ca0-9190-4674-af75-3fe93a97a8c8" containerName="revoke-keys" Oct 30 15:47:40.209865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.209848 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.227587 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.227551 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:47:40.227688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.227628 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:47:40.227688 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.227658 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:47:40.227819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.227698 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:47:40.229665 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.229639 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb"] Oct 30 15:47:40.284147 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.284116 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tckh\" (UniqueName: \"kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.284269 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.284160 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.284323 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.284289 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.385448 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.385422 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tckh\" (UniqueName: \"kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.385548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.385463 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.385548 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.385535 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.386218 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.386201 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.387761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.387739 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.410467 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.410421 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tckh\" (UniqueName: \"kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh\") pod \"maas-api-revoke-sub-3d4156f1-rpssb\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.523022 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.523002 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:40.821195 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:40.821167 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb"] Oct 30 15:47:40.822867 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:47:40.822837 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4618ede_74c1_4afa_bee0_944dadf83e09.slice/crio-bf8a9ea471d19d7a92b236a3d84524e453b82bdfe2016f49ff2fa1701bdd2de8 WatchSource:0}: Error finding container bf8a9ea471d19d7a92b236a3d84524e453b82bdfe2016f49ff2fa1701bdd2de8: Status 404 returned error can't find the container with id bf8a9ea471d19d7a92b236a3d84524e453b82bdfe2016f49ff2fa1701bdd2de8 Oct 30 15:47:41.192365 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:41.192315 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" event={"ID":"e4618ede-74c1-4afa-bee0-944dadf83e09","Type":"ContainerStarted","Data":"4215af207e4b08b01b482d6986c1446bf66ca35b3bbf02360120c157e8652814"} Oct 30 15:47:41.192559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:41.192396 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" event={"ID":"e4618ede-74c1-4afa-bee0-944dadf83e09","Type":"ContainerStarted","Data":"bf8a9ea471d19d7a92b236a3d84524e453b82bdfe2016f49ff2fa1701bdd2de8"} Oct 30 15:47:41.237529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:41.237486 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" podStartSLOduration=1.2374726329999999 podStartE2EDuration="1.237472633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:47:41.235742648 +0000 UTC m=+3565.194784052" watchObservedRunningTime="2026-10-30 15:47:41.237472633 +0000 UTC m=+3565.196514038" Oct 30 15:47:42.153442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:42.153405 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96464ca0-9190-4674-af75-3fe93a97a8c8" path="/var/lib/kubelet/pods/96464ca0-9190-4674-af75-3fe93a97a8c8/volumes" Oct 30 15:47:42.197991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:42.197967 6554 generic.go:356] "Generic (PLEG): container finished" podID="e4618ede-74c1-4afa-bee0-944dadf83e09" containerID="4215af207e4b08b01b482d6986c1446bf66ca35b3bbf02360120c157e8652814" exitCode=0 Oct 30 15:47:42.198124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:42.198050 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" event={"ID":"e4618ede-74c1-4afa-bee0-944dadf83e09","Type":"ContainerDied","Data":"4215af207e4b08b01b482d6986c1446bf66ca35b3bbf02360120c157e8652814"} Oct 30 15:47:43.332881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.332857 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:43.414007 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.413982 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca\") pod \"e4618ede-74c1-4afa-bee0-944dadf83e09\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " Oct 30 15:47:43.414165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.414040 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh\" (UniqueName: \"kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh\") pod \"e4618ede-74c1-4afa-bee0-944dadf83e09\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " Oct 30 15:47:43.414165 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.414132 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca\") pod \"e4618ede-74c1-4afa-bee0-944dadf83e09\" (UID: \"e4618ede-74c1-4afa-bee0-944dadf83e09\") " Oct 30 15:47:43.414545 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.414523 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca" pod "e4618ede-74c1-4afa-bee0-944dadf83e09" (UID: "e4618ede-74c1-4afa-bee0-944dadf83e09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:47:43.416059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.416039 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca" pod "e4618ede-74c1-4afa-bee0-944dadf83e09" (UID: "e4618ede-74c1-4afa-bee0-944dadf83e09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:47:43.416121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.416058 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh" pod "e4618ede-74c1-4afa-bee0-944dadf83e09" (UID: "e4618ede-74c1-4afa-bee0-944dadf83e09"). InnerVolumeSpecName "kube-api-access-4tckh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:47:43.515635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.515574 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:47:43.515635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.515601 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tckh\" (UniqueName: \"kubernetes.io/projected/e4618ede-74c1-4afa-bee0-944dadf83e09-kube-api-access-4tckh\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:47:43.515635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:43.515615 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4618ede-74c1-4afa-bee0-944dadf83e09-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:47:44.208424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:44.208393 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" Oct 30 15:47:44.208556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:44.208400 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb" event={"ID":"e4618ede-74c1-4afa-bee0-944dadf83e09","Type":"ContainerDied","Data":"bf8a9ea471d19d7a92b236a3d84524e453b82bdfe2016f49ff2fa1701bdd2de8"} Oct 30 15:47:44.208556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:47:44.208513 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf8a9ea471d19d7a92b236a3d84524e453b82bdfe2016f49ff2fa1701bdd2de8" Oct 30 15:48:09.201551 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:48:09.201511 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:48:12.035684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:12.035645 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84"] Oct 30 15:48:12.049453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:12.049421 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-f5x84"] Oct 30 15:48:12.153897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:12.153862 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c849e1f0-28fa-465f-a4f0-434d6e607c67" path="/var/lib/kubelet/pods/c849e1f0-28fa-465f-a4f0-434d6e607c67/volumes" Oct 30 15:48:18.498251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:18.498215 6554 scope.go:122] "RemoveContainer" containerID="47380c6497399dbfc931fd24947694f1f3834c58924f2790f38b6b16d1b6eacc" Oct 30 15:48:18.507181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:18.507160 6554 scope.go:122] "RemoveContainer" containerID="0c9d86798073c2e32b0cd14f8233e76f201fc437a88fb56a04014246d1d3b362" Oct 30 15:48:18.515405 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:18.515385 6554 scope.go:122] "RemoveContainer" containerID="34481886cd6802d661919e24c2243c8ba532cef55a99a28f9c69684180e4e9e1" Oct 30 15:48:39.200987 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:48:39.200949 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:48:47.409704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.409667 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7"] Oct 30 15:48:47.410305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.410078 6554 cpu_manager.go:414] "containerMap: removing container" podUID="e4618ede-74c1-4afa-bee0-944dadf83e09" containerName="revoke-keys" Oct 30 15:48:47.410305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.410091 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4618ede-74c1-4afa-bee0-944dadf83e09" containerName="revoke-keys" Oct 30 15:48:47.410305 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.410165 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4618ede-74c1-4afa-bee0-944dadf83e09" containerName="revoke-keys" Oct 30 15:48:47.413325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.413304 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.429732 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.429711 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:48:47.443660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.443635 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:48:47.443825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.443689 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:48:47.444059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.444041 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:48:47.446085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.446060 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7"] Oct 30 15:48:47.579438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.579405 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkss\" (UniqueName: \"kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.579595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.579465 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.579643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.579605 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.680170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.680099 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkss\" (UniqueName: \"kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.680170 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.680148 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.680396 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.680176 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.680959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.680936 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.682453 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.682432 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.705138 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.705108 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkss\" (UniqueName: \"kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss\") pod \"maas-api-revoke-sub-9c007c00-mzjc7\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:47.727251 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:47.727233 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:48.035120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:48.035094 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7"] Oct 30 15:48:48.036941 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:48:48.036914 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41215478_c4f3_462f_aad0_01bab6206436.slice/crio-342ea366e2a9f5ba9e437c948b327d79effefaba6a0110dd2cbcb984a8f1403d WatchSource:0}: Error finding container 342ea366e2a9f5ba9e437c948b327d79effefaba6a0110dd2cbcb984a8f1403d: Status 404 returned error can't find the container with id 342ea366e2a9f5ba9e437c948b327d79effefaba6a0110dd2cbcb984a8f1403d Oct 30 15:48:48.521216 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:48.521184 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" event={"ID":"41215478-c4f3-462f-aad0-01bab6206436","Type":"ContainerStarted","Data":"353e195665364eaa4437238f28d4a4807be126188ea38b67a0dc0e35ac2f2edb"} Oct 30 15:48:48.521661 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:48.521233 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" event={"ID":"41215478-c4f3-462f-aad0-01bab6206436","Type":"ContainerStarted","Data":"342ea366e2a9f5ba9e437c948b327d79effefaba6a0110dd2cbcb984a8f1403d"} Oct 30 15:48:48.565707 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:48.565662 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" podStartSLOduration=1.5656502799999998 podStartE2EDuration="1.56565028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:48.564482136 +0000 UTC m=+3632.523523541" watchObservedRunningTime="2026-10-30 15:48:48.56565028 +0000 UTC m=+3632.524691685" Oct 30 15:48:49.528199 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:49.528160 6554 generic.go:356] "Generic (PLEG): container finished" podID="41215478-c4f3-462f-aad0-01bab6206436" containerID="353e195665364eaa4437238f28d4a4807be126188ea38b67a0dc0e35ac2f2edb" exitCode=0 Oct 30 15:48:49.528704 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:49.528247 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" event={"ID":"41215478-c4f3-462f-aad0-01bab6206436","Type":"ContainerDied","Data":"353e195665364eaa4437238f28d4a4807be126188ea38b67a0dc0e35ac2f2edb"} Oct 30 15:48:50.659013 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.658988 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:50.818780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.818719 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss\" (UniqueName: \"kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss\") pod \"41215478-c4f3-462f-aad0-01bab6206436\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " Oct 30 15:48:50.818780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.818775 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca\") pod \"41215478-c4f3-462f-aad0-01bab6206436\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " Oct 30 15:48:50.818991 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.818814 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca\") pod \"41215478-c4f3-462f-aad0-01bab6206436\" (UID: \"41215478-c4f3-462f-aad0-01bab6206436\") " Oct 30 15:48:50.819201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.819175 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca" pod "41215478-c4f3-462f-aad0-01bab6206436" (UID: "41215478-c4f3-462f-aad0-01bab6206436"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:48:50.820929 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.820906 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss" pod "41215478-c4f3-462f-aad0-01bab6206436" (UID: "41215478-c4f3-462f-aad0-01bab6206436"). InnerVolumeSpecName "kube-api-access-7rkss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:48:50.821023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.820928 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca" pod "41215478-c4f3-462f-aad0-01bab6206436" (UID: "41215478-c4f3-462f-aad0-01bab6206436"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:48:50.920306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.920272 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkss\" (UniqueName: \"kubernetes.io/projected/41215478-c4f3-462f-aad0-01bab6206436-kube-api-access-7rkss\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:48:50.920306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.920305 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41215478-c4f3-462f-aad0-01bab6206436-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:48:50.920306 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:50.920314 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41215478-c4f3-462f-aad0-01bab6206436-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:48:51.542891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:51.542839 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" event={"ID":"41215478-c4f3-462f-aad0-01bab6206436","Type":"ContainerDied","Data":"342ea366e2a9f5ba9e437c948b327d79effefaba6a0110dd2cbcb984a8f1403d"} Oct 30 15:48:51.543104 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:51.542940 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="342ea366e2a9f5ba9e437c948b327d79effefaba6a0110dd2cbcb984a8f1403d" Oct 30 15:48:51.543173 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:51.543104 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7" Oct 30 15:48:58.673667 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.673633 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm"] Oct 30 15:48:58.674164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.673993 6554 cpu_manager.go:414] "containerMap: removing container" podUID="41215478-c4f3-462f-aad0-01bab6206436" containerName="revoke-keys" Oct 30 15:48:58.674164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.674005 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41215478-c4f3-462f-aad0-01bab6206436" containerName="revoke-keys" Oct 30 15:48:58.674164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.674101 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="41215478-c4f3-462f-aad0-01bab6206436" containerName="revoke-keys" Oct 30 15:48:58.677361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.677328 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.709010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.708983 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:48:58.709145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.709019 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:48:58.709145 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.709070 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:48:58.709267 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.709173 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:48:58.710922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.710895 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm"] Oct 30 15:48:58.790766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.790737 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54skf\" (UniqueName: \"kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.790891 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.790775 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.790953 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.790879 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.892033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.892006 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54skf\" (UniqueName: \"kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.892201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.892038 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.892201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.892063 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.892775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.892756 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.894241 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.894224 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.916464 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.916445 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54skf\" (UniqueName: \"kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:58.990775 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:58.990722 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:48:59.288672 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:59.288647 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm"] Oct 30 15:48:59.290414 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:48:59.290390 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9763097_448d_46f0_8fa3_de879dabedd8.slice/crio-1c773ea3fc051b368db13df91111c2988376f3460e568827e9865252e915b4ca WatchSource:0}: Error finding container 1c773ea3fc051b368db13df91111c2988376f3460e568827e9865252e915b4ca: Status 404 returned error can't find the container with id 1c773ea3fc051b368db13df91111c2988376f3460e568827e9865252e915b4ca Oct 30 15:48:59.584209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:59.584181 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" event={"ID":"c9763097-448d-46f0-8fa3-de879dabedd8","Type":"ContainerStarted","Data":"229be308198506373801af497fe06ce280a6b572d1dfacd412ee3ead357ad267"} Oct 30 15:48:59.584336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:59.584229 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" event={"ID":"c9763097-448d-46f0-8fa3-de879dabedd8","Type":"ContainerStarted","Data":"1c773ea3fc051b368db13df91111c2988376f3460e568827e9865252e915b4ca"} Oct 30 15:48:59.628444 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:48:59.628405 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" podStartSLOduration=1.628390921 podStartE2EDuration="1.628390921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:48:59.626822624 +0000 UTC m=+3643.585864030" watchObservedRunningTime="2026-10-30 15:48:59.628390921 +0000 UTC m=+3643.587432326" Oct 30 15:49:00.591026 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:00.590988 6554 generic.go:356] "Generic (PLEG): container finished" podID="c9763097-448d-46f0-8fa3-de879dabedd8" containerID="229be308198506373801af497fe06ce280a6b572d1dfacd412ee3ead357ad267" exitCode=0 Oct 30 15:49:00.591480 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:00.591077 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" event={"ID":"c9763097-448d-46f0-8fa3-de879dabedd8","Type":"ContainerDied","Data":"229be308198506373801af497fe06ce280a6b572d1dfacd412ee3ead357ad267"} Oct 30 15:49:01.736670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.736646 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:49:01.835072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.835045 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca\") pod \"c9763097-448d-46f0-8fa3-de879dabedd8\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " Oct 30 15:49:01.835223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.835166 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf\" (UniqueName: \"kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf\") pod \"c9763097-448d-46f0-8fa3-de879dabedd8\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " Oct 30 15:49:01.835223 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.835195 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca\") pod \"c9763097-448d-46f0-8fa3-de879dabedd8\" (UID: \"c9763097-448d-46f0-8fa3-de879dabedd8\") " Oct 30 15:49:01.835562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.835539 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca" pod "c9763097-448d-46f0-8fa3-de879dabedd8" (UID: "c9763097-448d-46f0-8fa3-de879dabedd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:49:01.837050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.837025 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca" pod "c9763097-448d-46f0-8fa3-de879dabedd8" (UID: "c9763097-448d-46f0-8fa3-de879dabedd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:49:01.837050 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.837023 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf" pod "c9763097-448d-46f0-8fa3-de879dabedd8" (UID: "c9763097-448d-46f0-8fa3-de879dabedd8"). InnerVolumeSpecName "kube-api-access-54skf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:49:01.936649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.936625 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:49:01.936649 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.936647 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54skf\" (UniqueName: \"kubernetes.io/projected/c9763097-448d-46f0-8fa3-de879dabedd8-kube-api-access-54skf\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:49:01.936782 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:01.936657 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9763097-448d-46f0-8fa3-de879dabedd8-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:49:02.603105 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:02.603078 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" Oct 30 15:49:02.603232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:02.603073 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm" event={"ID":"c9763097-448d-46f0-8fa3-de879dabedd8","Type":"ContainerDied","Data":"1c773ea3fc051b368db13df91111c2988376f3460e568827e9865252e915b4ca"} Oct 30 15:49:02.603232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:02.603212 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c773ea3fc051b368db13df91111c2988376f3460e568827e9865252e915b4ca" Oct 30 15:49:05.028813 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:05.028766 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v"] Oct 30 15:49:05.060542 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:05.060015 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-8kj4v"] Oct 30 15:49:06.153468 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:06.153433 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1" path="/var/lib/kubelet/pods/d21e3d76-0ab9-46b3-9716-0b8d4a0f24f1/volumes" Oct 30 15:49:09.201269 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:49:09.201203 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:49:12.292809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:12.292779 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm"] Oct 30 15:49:12.306024 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:12.305989 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c2454e-8bqrm"] Oct 30 15:49:14.153151 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:14.153116 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9763097-448d-46f0-8fa3-de879dabedd8" path="/var/lib/kubelet/pods/c9763097-448d-46f0-8fa3-de879dabedd8/volumes" Oct 30 15:49:18.545982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:18.545941 6554 scope.go:122] "RemoveContainer" containerID="b854493c3d2d8c502fa2cc10ca3785528d4a3e9d498819395d1a947894b131ce" Oct 30 15:49:20.721890 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.721854 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb"] Oct 30 15:49:20.735420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.735394 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-rpssb"] Oct 30 15:49:20.771530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.771497 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4"] Oct 30 15:49:20.771919 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.771900 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c9763097-448d-46f0-8fa3-de879dabedd8" containerName="revoke-keys" Oct 30 15:49:20.772004 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.771922 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9763097-448d-46f0-8fa3-de879dabedd8" containerName="revoke-keys" Oct 30 15:49:20.772084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.772070 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9763097-448d-46f0-8fa3-de879dabedd8" containerName="revoke-keys" Oct 30 15:49:20.775117 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.775097 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.794880 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.794856 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:49:20.795006 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.794961 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:49:20.795083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.795039 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:49:20.795083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.795074 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:49:20.795194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.795108 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4"] Oct 30 15:49:20.818986 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.818954 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrdp\" (UniqueName: \"kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.819127 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.819005 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.819127 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.819033 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.920001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.919973 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.920101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.920010 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.920148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.920130 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrdp\" (UniqueName: \"kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.920743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.920726 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.922420 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.922397 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:20.944727 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:20.944705 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrdp\" (UniqueName: \"kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp\") pod \"maas-api-revoke-sub-3d4156f1-tpwp4\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:21.088494 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:21.088443 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:21.386861 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:21.386834 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4"] Oct 30 15:49:21.388919 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:49:21.388889 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68dca1e_b163_4f25_97cb_f57a280dcfc0.slice/crio-6c7adee905bd086f4b28c623e7eb768e6080befc7b4005638a8060703175b4c0 WatchSource:0}: Error finding container 6c7adee905bd086f4b28c623e7eb768e6080befc7b4005638a8060703175b4c0: Status 404 returned error can't find the container with id 6c7adee905bd086f4b28c623e7eb768e6080befc7b4005638a8060703175b4c0 Oct 30 15:49:21.700635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:21.700607 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" event={"ID":"a68dca1e-b163-4f25-97cb-f57a280dcfc0","Type":"ContainerStarted","Data":"6a048cb2d19ad7d4a6ae26a7a536d2fdf1227a1f4b299aad7c0c1e60d78ab725"} Oct 30 15:49:21.700741 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:21.700657 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" event={"ID":"a68dca1e-b163-4f25-97cb-f57a280dcfc0","Type":"ContainerStarted","Data":"6c7adee905bd086f4b28c623e7eb768e6080befc7b4005638a8060703175b4c0"} Oct 30 15:49:21.744581 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:21.744538 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" podStartSLOduration=1.744525669 podStartE2EDuration="1.744525669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:49:21.743447848 +0000 UTC m=+3665.702489254" watchObservedRunningTime="2026-10-30 15:49:21.744525669 +0000 UTC m=+3665.703567074" Oct 30 15:49:22.152865 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:22.152836 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4618ede-74c1-4afa-bee0-944dadf83e09" path="/var/lib/kubelet/pods/e4618ede-74c1-4afa-bee0-944dadf83e09/volumes" Oct 30 15:49:22.707809 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:22.707776 6554 generic.go:356] "Generic (PLEG): container finished" podID="a68dca1e-b163-4f25-97cb-f57a280dcfc0" containerID="6a048cb2d19ad7d4a6ae26a7a536d2fdf1227a1f4b299aad7c0c1e60d78ab725" exitCode=0 Oct 30 15:49:22.707964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:22.707876 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" event={"ID":"a68dca1e-b163-4f25-97cb-f57a280dcfc0","Type":"ContainerDied","Data":"6a048cb2d19ad7d4a6ae26a7a536d2fdf1227a1f4b299aad7c0c1e60d78ab725"} Oct 30 15:49:23.837206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.837182 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:23.945643 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.945611 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp\" (UniqueName: \"kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp\") pod \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " Oct 30 15:49:23.945819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.945747 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca\") pod \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " Oct 30 15:49:23.945819 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.945774 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca\") pod \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\" (UID: \"a68dca1e-b163-4f25-97cb-f57a280dcfc0\") " Oct 30 15:49:23.946163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.946134 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca" pod "a68dca1e-b163-4f25-97cb-f57a280dcfc0" (UID: "a68dca1e-b163-4f25-97cb-f57a280dcfc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:49:23.947769 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.947741 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp" pod "a68dca1e-b163-4f25-97cb-f57a280dcfc0" (UID: "a68dca1e-b163-4f25-97cb-f57a280dcfc0"). InnerVolumeSpecName "kube-api-access-bnrdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:49:23.947876 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:23.947768 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca" pod "a68dca1e-b163-4f25-97cb-f57a280dcfc0" (UID: "a68dca1e-b163-4f25-97cb-f57a280dcfc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:49:24.047217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:24.047145 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrdp\" (UniqueName: \"kubernetes.io/projected/a68dca1e-b163-4f25-97cb-f57a280dcfc0-kube-api-access-bnrdp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:49:24.047217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:24.047171 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:49:24.047217 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:24.047183 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a68dca1e-b163-4f25-97cb-f57a280dcfc0-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:49:24.719712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:24.719676 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" event={"ID":"a68dca1e-b163-4f25-97cb-f57a280dcfc0","Type":"ContainerDied","Data":"6c7adee905bd086f4b28c623e7eb768e6080befc7b4005638a8060703175b4c0"} Oct 30 15:49:24.719901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:24.719722 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4" Oct 30 15:49:24.719901 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:24.719734 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c7adee905bd086f4b28c623e7eb768e6080befc7b4005638a8060703175b4c0" Oct 30 15:49:29.029863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:29.029821 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm"] Oct 30 15:49:29.043518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:29.043490 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-m6cbm"] Oct 30 15:49:30.152897 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:30.152866 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b76ef6-7342-4597-b034-a35a453f791d" path="/var/lib/kubelet/pods/c0b76ef6-7342-4597-b034-a35a453f791d/volumes" Oct 30 15:49:39.202052 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:49:39.202010 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:49:40.027635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:40.027592 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx"] Oct 30 15:49:40.041201 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:40.041169 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-55bbx"] Oct 30 15:49:40.152742 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:49:40.152712 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c" path="/var/lib/kubelet/pods/9dc2fbc3-8a08-4d54-9b2f-3b4427d4cb4c/volumes" Oct 30 15:50:09.200953 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:50:09.200875 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:50:18.572878 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:18.572845 6554 scope.go:122] "RemoveContainer" containerID="27af8cda9f74651b0167b6a3b08aab7c5c0b464e83b8f5449a2745addc7041ae" Oct 30 15:50:24.032963 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:24.032919 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r"] Oct 30 15:50:24.044526 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:24.044496 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kzh4r"] Oct 30 15:50:24.152279 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:24.152248 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82" path="/var/lib/kubelet/pods/ca5fe0ef-94d2-47e9-80a0-4fbbd6ae6d82/volumes" Oct 30 15:50:39.200119 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:50:39.200079 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:50:45.031443 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:45.031406 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh"] Oct 30 15:50:45.044821 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:45.044797 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-bksbh"] Oct 30 15:50:46.153903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:50:46.153868 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50317320-f011-4db5-98d9-5836b6dc601b" path="/var/lib/kubelet/pods/50317320-f011-4db5-98d9-5836b6dc601b/volumes" Oct 30 15:51:07.031530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:07.031480 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4"] Oct 30 15:51:07.044559 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:07.044529 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hfcf4"] Oct 30 15:51:08.154817 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:08.154768 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8fee90-3c09-4e5c-88b0-55a1f65cfa09" path="/var/lib/kubelet/pods/9d8fee90-3c09-4e5c-88b0-55a1f65cfa09/volumes" Oct 30 15:51:09.245798 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:51:09.245757 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:51:18.595427 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:18.595390 6554 scope.go:122] "RemoveContainer" containerID="5b51cf6c1a5e07fb053a307e4ec86d31b3fcdc2b9b959fee033c67aaf09fa910" Oct 30 15:51:18.604849 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:18.604817 6554 scope.go:122] "RemoveContainer" containerID="e196bf6d0d193fceafdd59d991a323f51d46557c45351dd083df90e2df6293cf" Oct 30 15:51:27.030885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:27.030650 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q"] Oct 30 15:51:27.044249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:27.044222 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vx26q"] Oct 30 15:51:28.045361 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.045316 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84"] Oct 30 15:51:28.045808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.045714 6554 cpu_manager.go:414] "containerMap: removing container" podUID="a68dca1e-b163-4f25-97cb-f57a280dcfc0" containerName="revoke-keys" Oct 30 15:51:28.045808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.045727 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68dca1e-b163-4f25-97cb-f57a280dcfc0" containerName="revoke-keys" Oct 30 15:51:28.045808 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.045809 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68dca1e-b163-4f25-97cb-f57a280dcfc0" containerName="revoke-keys" Oct 30 15:51:28.048884 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.048867 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.067611 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.067589 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:51:28.081330 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.081306 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:51:28.081463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.081390 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:51:28.081463 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.081391 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:51:28.082932 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.082912 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84"] Oct 30 15:51:28.152083 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.152058 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b97c9a-f5c1-445b-9e7b-2b3df688db57" path="/var/lib/kubelet/pods/69b97c9a-f5c1-445b-9e7b-2b3df688db57/volumes" Oct 30 15:51:28.229832 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.229803 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.229952 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.229861 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.230027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.229951 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z4jz\" (UniqueName: \"kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.331138 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.331084 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.331252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.331144 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.331252 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.331178 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z4jz\" (UniqueName: \"kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.332063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.332041 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.333622 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.333597 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.372730 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.368971 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z4jz\" (UniqueName: \"kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz\") pod \"maas-api-revoke-sub-d2bc93bc-jbr84\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.661270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.661247 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:28.959842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:28.959819 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84"] Oct 30 15:51:28.962027 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:51:28.961992 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e1d5ea_0e63_4d82_b07b_fb8a772f1114.slice/crio-3217063808f257cba9faff161e01cda537a121bfee94941e3cd881e9a55384cd WatchSource:0}: Error finding container 3217063808f257cba9faff161e01cda537a121bfee94941e3cd881e9a55384cd: Status 404 returned error can't find the container with id 3217063808f257cba9faff161e01cda537a121bfee94941e3cd881e9a55384cd Oct 30 15:51:29.441023 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:29.440993 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" event={"ID":"38e1d5ea-0e63-4d82-b07b-fb8a772f1114","Type":"ContainerStarted","Data":"2a2d9c872b66d7c2e6cc646be33dcc75808d8c2712dca837235986b928ad378a"} Oct 30 15:51:29.441329 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:29.441047 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" event={"ID":"38e1d5ea-0e63-4d82-b07b-fb8a772f1114","Type":"ContainerStarted","Data":"3217063808f257cba9faff161e01cda537a121bfee94941e3cd881e9a55384cd"} Oct 30 15:51:29.485204 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:29.485158 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" podStartSLOduration=1.485146362 podStartE2EDuration="1.485146362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:51:29.483671504 +0000 UTC m=+3793.442712908" watchObservedRunningTime="2026-10-30 15:51:29.485146362 +0000 UTC m=+3793.444187763" Oct 30 15:51:30.447792 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:30.447752 6554 generic.go:356] "Generic (PLEG): container finished" podID="38e1d5ea-0e63-4d82-b07b-fb8a772f1114" containerID="2a2d9c872b66d7c2e6cc646be33dcc75808d8c2712dca837235986b928ad378a" exitCode=0 Oct 30 15:51:30.448280 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:30.447838 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" event={"ID":"38e1d5ea-0e63-4d82-b07b-fb8a772f1114","Type":"ContainerDied","Data":"2a2d9c872b66d7c2e6cc646be33dcc75808d8c2712dca837235986b928ad378a"} Oct 30 15:51:31.591980 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.591950 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:31.762997 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.762935 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz\" (UniqueName: \"kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz\") pod \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " Oct 30 15:51:31.763144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.763001 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca\") pod \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " Oct 30 15:51:31.763144 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.763100 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca\") pod \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\" (UID: \"38e1d5ea-0e63-4d82-b07b-fb8a772f1114\") " Oct 30 15:51:31.763364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.763323 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca" pod "38e1d5ea-0e63-4d82-b07b-fb8a772f1114" (UID: "38e1d5ea-0e63-4d82-b07b-fb8a772f1114"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:51:31.763482 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.763467 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:51:31.765101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.765080 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz" pod "38e1d5ea-0e63-4d82-b07b-fb8a772f1114" (UID: "38e1d5ea-0e63-4d82-b07b-fb8a772f1114"). InnerVolumeSpecName "kube-api-access-9z4jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:51:31.765191 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.765148 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca" pod "38e1d5ea-0e63-4d82-b07b-fb8a772f1114" (UID: "38e1d5ea-0e63-4d82-b07b-fb8a772f1114"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:51:31.864504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.864481 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:51:31.864504 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:31.864504 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z4jz\" (UniqueName: \"kubernetes.io/projected/38e1d5ea-0e63-4d82-b07b-fb8a772f1114-kube-api-access-9z4jz\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:51:32.459720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:32.459655 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" Oct 30 15:51:32.459720 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:32.459691 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84" event={"ID":"38e1d5ea-0e63-4d82-b07b-fb8a772f1114","Type":"ContainerDied","Data":"3217063808f257cba9faff161e01cda537a121bfee94941e3cd881e9a55384cd"} Oct 30 15:51:32.459916 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:32.459742 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3217063808f257cba9faff161e01cda537a121bfee94941e3cd881e9a55384cd" Oct 30 15:51:39.200766 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:51:39.200731 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:51:48.029820 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:48.029784 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd"] Oct 30 15:51:48.043275 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:48.043247 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-dnhnd"] Oct 30 15:51:48.153074 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:51:48.153040 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ff11a0-7083-4251-8660-934e24e9c2d5" path="/var/lib/kubelet/pods/a0ff11a0-7083-4251-8660-934e24e9c2d5/volumes" Oct 30 15:52:09.201334 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:52:09.201292 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:52:18.582291 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.582255 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf"] Oct 30 15:52:18.582749 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.582729 6554 cpu_manager.go:414] "containerMap: removing container" podUID="38e1d5ea-0e63-4d82-b07b-fb8a772f1114" containerName="revoke-keys" Oct 30 15:52:18.582796 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.582750 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e1d5ea-0e63-4d82-b07b-fb8a772f1114" containerName="revoke-keys" Oct 30 15:52:18.582885 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.582871 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e1d5ea-0e63-4d82-b07b-fb8a772f1114" containerName="revoke-keys" Oct 30 15:52:18.586580 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.586560 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.617664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.617639 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:52:18.617664 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.617639 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:52:18.617838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.617704 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:52:18.617838 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.617746 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:52:18.620140 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.620117 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf"] Oct 30 15:52:18.629178 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.629160 6554 scope.go:122] "RemoveContainer" containerID="50495df48b230e2dbc7de2dccd993f880e2071e62ecf2fc66294a78b22946d78" Oct 30 15:52:18.637899 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.637880 6554 scope.go:122] "RemoveContainer" containerID="2d44cd523bb5f735e9475252ea7313cb6a1f44d8333fb792a2bb16a14ce8f176" Oct 30 15:52:18.645787 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.645766 6554 scope.go:122] "RemoveContainer" containerID="547a416f963a1e512ae9d9d753eb46ab01f851d59066839cae75c0e01990a53a" Oct 30 15:52:18.711737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.711707 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxwc\" (UniqueName: \"kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.711915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.711758 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.711915 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.711858 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.812651 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.812629 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.812767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.812671 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.812767 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.812752 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxwc\" (UniqueName: \"kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.813265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.813240 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.814881 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.814863 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.838111 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.838058 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxwc\" (UniqueName: \"kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc\") pod \"maas-api-revoke-sub-5d054765-wl5sf\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:18.898795 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:18.898773 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:19.219931 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:19.219904 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf"] Oct 30 15:52:19.222423 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:52:19.222397 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ea4f16_d76d_4095_846c_7a5e76f8e9af.slice/crio-5f56cc381fdfdf439238efa0075226e0d120b59cfb1aec9346aff067b91d2efc WatchSource:0}: Error finding container 5f56cc381fdfdf439238efa0075226e0d120b59cfb1aec9346aff067b91d2efc: Status 404 returned error can't find the container with id 5f56cc381fdfdf439238efa0075226e0d120b59cfb1aec9346aff067b91d2efc Oct 30 15:52:19.710533 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:19.710500 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" event={"ID":"c1ea4f16-d76d-4095-846c-7a5e76f8e9af","Type":"ContainerStarted","Data":"5f575f1cec2048aa0f79c5ab4c5a8d066b286056e6b95b9c8e5311ffac90594e"} Oct 30 15:52:19.710862 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:19.710557 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" event={"ID":"c1ea4f16-d76d-4095-846c-7a5e76f8e9af","Type":"ContainerStarted","Data":"5f56cc381fdfdf439238efa0075226e0d120b59cfb1aec9346aff067b91d2efc"} Oct 30 15:52:19.754524 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:19.754431 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" podStartSLOduration=1.7544143779999999 podStartE2EDuration="1.754414378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:19.753715444 +0000 UTC m=+3843.712756851" watchObservedRunningTime="2026-10-30 15:52:19.754414378 +0000 UTC m=+3843.713455785" Oct 30 15:52:20.716724 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:20.716689 6554 generic.go:356] "Generic (PLEG): container finished" podID="c1ea4f16-d76d-4095-846c-7a5e76f8e9af" containerID="5f575f1cec2048aa0f79c5ab4c5a8d066b286056e6b95b9c8e5311ffac90594e" exitCode=0 Oct 30 15:52:20.717141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:20.716780 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" event={"ID":"c1ea4f16-d76d-4095-846c-7a5e76f8e9af","Type":"ContainerDied","Data":"5f575f1cec2048aa0f79c5ab4c5a8d066b286056e6b95b9c8e5311ffac90594e"} Oct 30 15:52:21.866197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:21.866171 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:22.043016 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.042942 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca\") pod \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " Oct 30 15:52:22.043174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.043051 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca\") pod \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " Oct 30 15:52:22.043174 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.043077 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc\" (UniqueName: \"kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc\") pod \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\" (UID: \"c1ea4f16-d76d-4095-846c-7a5e76f8e9af\") " Oct 30 15:52:22.043401 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.043375 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca" pod "c1ea4f16-d76d-4095-846c-7a5e76f8e9af" (UID: "c1ea4f16-d76d-4095-846c-7a5e76f8e9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:52:22.045122 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.045095 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca" pod "c1ea4f16-d76d-4095-846c-7a5e76f8e9af" (UID: "c1ea4f16-d76d-4095-846c-7a5e76f8e9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:52:22.045240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.045214 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc" pod "c1ea4f16-d76d-4095-846c-7a5e76f8e9af" (UID: "c1ea4f16-d76d-4095-846c-7a5e76f8e9af"). InnerVolumeSpecName "kube-api-access-gcxwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:52:22.143912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.143872 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:52:22.143912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.143910 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxwc\" (UniqueName: \"kubernetes.io/projected/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-kube-api-access-gcxwc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:52:22.144060 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.143926 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1ea4f16-d76d-4095-846c-7a5e76f8e9af-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:52:22.736086 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.736053 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" Oct 30 15:52:22.736249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.736096 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf" event={"ID":"c1ea4f16-d76d-4095-846c-7a5e76f8e9af","Type":"ContainerDied","Data":"5f56cc381fdfdf439238efa0075226e0d120b59cfb1aec9346aff067b91d2efc"} Oct 30 15:52:22.736249 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:22.736190 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f56cc381fdfdf439238efa0075226e0d120b59cfb1aec9346aff067b91d2efc" Oct 30 15:52:29.732541 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.732511 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g"] Oct 30 15:52:29.733010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.732887 6554 cpu_manager.go:414] "containerMap: removing container" podUID="c1ea4f16-d76d-4095-846c-7a5e76f8e9af" containerName="revoke-keys" Oct 30 15:52:29.733010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.732897 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1ea4f16-d76d-4095-846c-7a5e76f8e9af" containerName="revoke-keys" Oct 30 15:52:29.733010 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.732957 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1ea4f16-d76d-4095-846c-7a5e76f8e9af" containerName="revoke-keys" Oct 30 15:52:29.736147 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.736124 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.755072 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.755049 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:52:29.755221 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.755183 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:52:29.768903 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.768878 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:52:29.769085 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.768932 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:52:29.773172 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.773151 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g"] Oct 30 15:52:29.810942 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.810916 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.811061 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.810968 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.811133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.811091 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jppnc\" (UniqueName: \"kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.912493 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.912469 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.912589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.912519 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.912684 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.912665 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jppnc\" (UniqueName: \"kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.913101 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.913078 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.914802 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.914781 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:29.937697 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:29.937673 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jppnc\" (UniqueName: \"kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc\") pod \"maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:30.048451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:30.048404 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:30.357319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:30.357292 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g"] Oct 30 15:52:30.359587 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:52:30.359555 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26da7598_6001_42ae_a5bc_a86efde095a4.slice/crio-d0faead309a7fb2191a8cdf5b4e816e3ddc2469b19642588d264bd580f85bf76 WatchSource:0}: Error finding container d0faead309a7fb2191a8cdf5b4e816e3ddc2469b19642588d264bd580f85bf76: Status 404 returned error can't find the container with id d0faead309a7fb2191a8cdf5b4e816e3ddc2469b19642588d264bd580f85bf76 Oct 30 15:52:30.776739 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:30.776707 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" event={"ID":"26da7598-6001-42ae-a5bc-a86efde095a4","Type":"ContainerStarted","Data":"7a29f3b16004de27c738a7525c1bc7369d6f111eb90f0663274d9e2438e59b51"} Oct 30 15:52:30.777102 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:30.776761 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" event={"ID":"26da7598-6001-42ae-a5bc-a86efde095a4","Type":"ContainerStarted","Data":"d0faead309a7fb2191a8cdf5b4e816e3ddc2469b19642588d264bd580f85bf76"} Oct 30 15:52:30.822556 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:30.822509 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" podStartSLOduration=1.822496356 podStartE2EDuration="1.822496356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:52:30.821774152 +0000 UTC m=+3854.780815569" watchObservedRunningTime="2026-10-30 15:52:30.822496356 +0000 UTC m=+3854.781537762" Oct 30 15:52:31.783324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:31.783295 6554 generic.go:356] "Generic (PLEG): container finished" podID="26da7598-6001-42ae-a5bc-a86efde095a4" containerID="7a29f3b16004de27c738a7525c1bc7369d6f111eb90f0663274d9e2438e59b51" exitCode=0 Oct 30 15:52:31.783719 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:31.783381 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" event={"ID":"26da7598-6001-42ae-a5bc-a86efde095a4","Type":"ContainerDied","Data":"7a29f3b16004de27c738a7525c1bc7369d6f111eb90f0663274d9e2438e59b51"} Oct 30 15:52:32.914438 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.914416 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:32.936927 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.936901 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca\") pod \"26da7598-6001-42ae-a5bc-a86efde095a4\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " Oct 30 15:52:32.937040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.936979 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca\") pod \"26da7598-6001-42ae-a5bc-a86efde095a4\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " Oct 30 15:52:32.937040 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.937020 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc\" (UniqueName: \"kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc\") pod \"26da7598-6001-42ae-a5bc-a86efde095a4\" (UID: \"26da7598-6001-42ae-a5bc-a86efde095a4\") " Oct 30 15:52:32.937316 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.937293 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca" pod "26da7598-6001-42ae-a5bc-a86efde095a4" (UID: "26da7598-6001-42ae-a5bc-a86efde095a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:52:32.938985 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.938959 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca" pod "26da7598-6001-42ae-a5bc-a86efde095a4" (UID: "26da7598-6001-42ae-a5bc-a86efde095a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:52:32.939065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:32.939029 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc" pod "26da7598-6001-42ae-a5bc-a86efde095a4" (UID: "26da7598-6001-42ae-a5bc-a86efde095a4"). InnerVolumeSpecName "kube-api-access-jppnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:52:33.038479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:33.038452 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:52:33.038479 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:33.038476 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jppnc\" (UniqueName: \"kubernetes.io/projected/26da7598-6001-42ae-a5bc-a86efde095a4-kube-api-access-jppnc\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:52:33.038617 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:33.038487 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26da7598-6001-42ae-a5bc-a86efde095a4-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:52:33.795084 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:33.795051 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" Oct 30 15:52:33.795250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:33.795050 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g" event={"ID":"26da7598-6001-42ae-a5bc-a86efde095a4","Type":"ContainerDied","Data":"d0faead309a7fb2191a8cdf5b4e816e3ddc2469b19642588d264bd580f85bf76"} Oct 30 15:52:33.795250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:33.795201 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0faead309a7fb2191a8cdf5b4e816e3ddc2469b19642588d264bd580f85bf76" Oct 30 15:52:39.199666 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:52:39.199630 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:52:42.809157 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:42.809126 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g"] Oct 30 15:52:42.825623 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:42.825582 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-22f590-8pl6g"] Oct 30 15:52:44.158912 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:52:44.158878 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26da7598-6001-42ae-a5bc-a86efde095a4" path="/var/lib/kubelet/pods/26da7598-6001-42ae-a5bc-a86efde095a4/volumes" Oct 30 15:53:09.201000 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:53:09.200962 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:53:18.671835 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:18.671799 6554 scope.go:122] "RemoveContainer" containerID="98f8a44d6080706aa9279b63b961bc83587bcf10b18b68ad836d53e0bec15e6a" Oct 30 15:53:18.681283 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:18.681262 6554 scope.go:122] "RemoveContainer" containerID="b489b0d83f6eae3cb8cc2602b2dd4be1bb10c2cbb58662cb4b37769a466342c3" Oct 30 15:53:18.689270 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:18.689247 6554 scope.go:122] "RemoveContainer" containerID="651753ce1529e6de77a6c76541fd1acda14db5c9b6a6d37df0b56bfbfda3e018" Oct 30 15:53:18.697243 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:18.697223 6554 scope.go:122] "RemoveContainer" containerID="4dda96161844de6bfeaa8d2eca5db2902905ff98cb991f906ac90031a287f926" Oct 30 15:53:28.185230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.185195 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg"] Oct 30 15:53:28.185666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.185607 6554 cpu_manager.go:414] "containerMap: removing container" podUID="26da7598-6001-42ae-a5bc-a86efde095a4" containerName="revoke-keys" Oct 30 15:53:28.185666 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.185619 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26da7598-6001-42ae-a5bc-a86efde095a4" containerName="revoke-keys" Oct 30 15:53:28.185781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.185696 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="26da7598-6001-42ae-a5bc-a86efde095a4" containerName="revoke-keys" Oct 30 15:53:28.188760 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.188742 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.207013 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.206992 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:53:28.220512 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.220488 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:53:28.220766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.220729 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:53:28.220766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.220759 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:53:28.223367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.223327 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg"] Oct 30 15:53:28.346562 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.346530 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.346705 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.346689 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhxp\" (UniqueName: \"kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.346781 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.346736 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.447662 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.447603 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.447786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.447675 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhxp\" (UniqueName: \"kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.447786 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.447706 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.448293 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.448272 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.450194 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.450170 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.472803 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.472775 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhxp\" (UniqueName: \"kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp\") pod \"maas-api-revoke-sub-452b6d7b-xt5rg\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.501592 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.501560 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:28.799638 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:28.799613 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg"] Oct 30 15:53:28.802138 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:53:28.802109 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ba3fd2_8e47_4656_8899_063bbb81b052.slice/crio-64ff0428bd76578361420320ab52f5d95fc644c2d25d2f581f4208b4508397d9 WatchSource:0}: Error finding container 64ff0428bd76578361420320ab52f5d95fc644c2d25d2f581f4208b4508397d9: Status 404 returned error can't find the container with id 64ff0428bd76578361420320ab52f5d95fc644c2d25d2f581f4208b4508397d9 Oct 30 15:53:29.089209 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:29.089185 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" event={"ID":"28ba3fd2-8e47-4656-8899-063bbb81b052","Type":"ContainerStarted","Data":"714d96795cac76d10b6fb2c73a3f30ee887b42d137ce691636dbcaed5951f261"} Oct 30 15:53:29.089324 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:29.089233 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" event={"ID":"28ba3fd2-8e47-4656-8899-063bbb81b052","Type":"ContainerStarted","Data":"64ff0428bd76578361420320ab52f5d95fc644c2d25d2f581f4208b4508397d9"} Oct 30 15:53:29.135029 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:29.134978 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" podStartSLOduration=1.134966075 podStartE2EDuration="1.134966075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:53:29.132451853 +0000 UTC m=+3913.091493259" watchObservedRunningTime="2026-10-30 15:53:29.134966075 +0000 UTC m=+3913.094007525" Oct 30 15:53:30.098164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:30.098123 6554 generic.go:356] "Generic (PLEG): container finished" podID="28ba3fd2-8e47-4656-8899-063bbb81b052" containerID="714d96795cac76d10b6fb2c73a3f30ee887b42d137ce691636dbcaed5951f261" exitCode=0 Oct 30 15:53:30.098695 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:30.098239 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" event={"ID":"28ba3fd2-8e47-4656-8899-063bbb81b052","Type":"ContainerDied","Data":"714d96795cac76d10b6fb2c73a3f30ee887b42d137ce691636dbcaed5951f261"} Oct 30 15:53:31.451308 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.451277 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:31.604384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.603648 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca\") pod \"28ba3fd2-8e47-4656-8899-063bbb81b052\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " Oct 30 15:53:31.604384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.603734 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca\") pod \"28ba3fd2-8e47-4656-8899-063bbb81b052\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " Oct 30 15:53:31.604384 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.603813 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp\" (UniqueName: \"kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp\") pod \"28ba3fd2-8e47-4656-8899-063bbb81b052\" (UID: \"28ba3fd2-8e47-4656-8899-063bbb81b052\") " Oct 30 15:53:31.614617 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.614578 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca" pod "28ba3fd2-8e47-4656-8899-063bbb81b052" (UID: "28ba3fd2-8e47-4656-8899-063bbb81b052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:31.624518 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.624474 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp" pod "28ba3fd2-8e47-4656-8899-063bbb81b052" (UID: "28ba3fd2-8e47-4656-8899-063bbb81b052"). InnerVolumeSpecName "kube-api-access-srhxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:31.627882 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.627840 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca" pod "28ba3fd2-8e47-4656-8899-063bbb81b052" (UID: "28ba3fd2-8e47-4656-8899-063bbb81b052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:31.706400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.705934 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:53:31.706400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.705968 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28ba3fd2-8e47-4656-8899-063bbb81b052-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:53:31.706400 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:31.705984 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhxp\" (UniqueName: \"kubernetes.io/projected/28ba3fd2-8e47-4656-8899-063bbb81b052-kube-api-access-srhxp\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:53:32.131414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:32.130493 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" event={"ID":"28ba3fd2-8e47-4656-8899-063bbb81b052","Type":"ContainerDied","Data":"64ff0428bd76578361420320ab52f5d95fc644c2d25d2f581f4208b4508397d9"} Oct 30 15:53:32.131414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:32.130573 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64ff0428bd76578361420320ab52f5d95fc644c2d25d2f581f4208b4508397d9" Oct 30 15:53:32.131414 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:32.130631 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-xt5rg" Oct 30 15:53:39.199772 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:53:39.199733 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:53:49.466989 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.466954 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh"] Oct 30 15:53:49.467608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.467417 6554 cpu_manager.go:414] "containerMap: removing container" podUID="28ba3fd2-8e47-4656-8899-063bbb81b052" containerName="revoke-keys" Oct 30 15:53:49.467608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.467433 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ba3fd2-8e47-4656-8899-063bbb81b052" containerName="revoke-keys" Oct 30 15:53:49.467608 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.467499 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ba3fd2-8e47-4656-8899-063bbb81b052" containerName="revoke-keys" Oct 30 15:53:49.471048 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.471029 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.493315 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.493283 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh"] Oct 30 15:53:49.505090 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.505062 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:53:49.505233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.505072 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 15:53:49.505233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.505128 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:53:49.505233 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.505213 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4zs7x\"" Oct 30 15:53:49.533338 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.533309 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.533486 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.533456 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdc2f\" (UniqueName: \"kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.533597 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.533567 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.634454 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.634423 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdc2f\" (UniqueName: \"kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.634607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.634470 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.634607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.634572 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.635198 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.635172 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.636791 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.636770 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.659567 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.659542 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdc2f\" (UniqueName: \"kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f\") pod \"maas-api-revoke-sub-3e31a506-f9zzh\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:49.784265 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:49.784186 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:50.110057 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:50.109968 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh"] Oct 30 15:53:50.113199 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:53:50.113166 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb78ed19_043e_49f8_aef5_746fd6cd03a1.slice/crio-60f6606a5b44b8ecdb2b6f7e8a8b1641fee27d5511cf8319c056c0238c8387d3 WatchSource:0}: Error finding container 60f6606a5b44b8ecdb2b6f7e8a8b1641fee27d5511cf8319c056c0238c8387d3: Status 404 returned error can't find the container with id 60f6606a5b44b8ecdb2b6f7e8a8b1641fee27d5511cf8319c056c0238c8387d3 Oct 30 15:53:50.252019 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:50.251977 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" event={"ID":"fb78ed19-043e-49f8-aef5-746fd6cd03a1","Type":"ContainerStarted","Data":"60f6606a5b44b8ecdb2b6f7e8a8b1641fee27d5511cf8319c056c0238c8387d3"} Oct 30 15:53:51.037364 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:51.037286 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7"] Oct 30 15:53:51.040152 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:51.040100 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c007c00-mzjc7"] Oct 30 15:53:51.284136 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:51.284049 6554 generic.go:356] "Generic (PLEG): container finished" podID="fb78ed19-043e-49f8-aef5-746fd6cd03a1" containerID="ad6a4dad3cb1a82c096797f97e2260838be76ca9ed56dec13dc96a6f0a6fe981" exitCode=0 Oct 30 15:53:51.284411 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:51.284385 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" event={"ID":"fb78ed19-043e-49f8-aef5-746fd6cd03a1","Type":"ContainerDied","Data":"ad6a4dad3cb1a82c096797f97e2260838be76ca9ed56dec13dc96a6f0a6fe981"} Oct 30 15:53:52.159373 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.153629 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41215478-c4f3-462f-aad0-01bab6206436" path="/var/lib/kubelet/pods/41215478-c4f3-462f-aad0-01bab6206436/volumes" Oct 30 15:53:52.447863 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.447836 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:52.593065 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.593033 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca\") pod \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " Oct 30 15:53:52.593230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.593124 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca\") pod \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " Oct 30 15:53:52.593230 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.593155 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f\" (UniqueName: \"kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f\") pod \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\" (UID: \"fb78ed19-043e-49f8-aef5-746fd6cd03a1\") " Oct 30 15:53:52.593421 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.593400 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca" pod "fb78ed19-043e-49f8-aef5-746fd6cd03a1" (UID: "fb78ed19-043e-49f8-aef5-746fd6cd03a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:53:52.593514 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.593498 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-openshift-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:53:52.595232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.595206 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca" pod "fb78ed19-043e-49f8-aef5-746fd6cd03a1" (UID: "fb78ed19-043e-49f8-aef5-746fd6cd03a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:53:52.595380 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.595247 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f" pod "fb78ed19-043e-49f8-aef5-746fd6cd03a1" (UID: "fb78ed19-043e-49f8-aef5-746fd6cd03a1"). InnerVolumeSpecName "kube-api-access-pdc2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:53:52.694483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.694424 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb78ed19-043e-49f8-aef5-746fd6cd03a1-maas-api-xks-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:53:52.694483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:52.694448 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdc2f\" (UniqueName: \"kubernetes.io/projected/fb78ed19-043e-49f8-aef5-746fd6cd03a1-kube-api-access-pdc2f\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:53:53.301187 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:53.301156 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" Oct 30 15:53:53.301619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:53.301201 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-f9zzh" event={"ID":"fb78ed19-043e-49f8-aef5-746fd6cd03a1","Type":"ContainerDied","Data":"60f6606a5b44b8ecdb2b6f7e8a8b1641fee27d5511cf8319c056c0238c8387d3"} Oct 30 15:53:53.301619 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:53:53.301245 6554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60f6606a5b44b8ecdb2b6f7e8a8b1641fee27d5511cf8319c056c0238c8387d3" Oct 30 15:54:09.210660 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:54:09.210615 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:54:18.726133 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:18.726097 6554 scope.go:122] "RemoveContainer" containerID="4215af207e4b08b01b482d6986c1446bf66ca35b3bbf02360120c157e8652814" Oct 30 15:54:22.422457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.422421 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4"] Oct 30 15:54:22.423366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.422959 6554 cpu_manager.go:414] "containerMap: removing container" podUID="fb78ed19-043e-49f8-aef5-746fd6cd03a1" containerName="revoke-keys" Oct 30 15:54:22.423366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.422979 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb78ed19-043e-49f8-aef5-746fd6cd03a1" containerName="revoke-keys" Oct 30 15:54:22.423366 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.423119 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb78ed19-043e-49f8-aef5-746fd6cd03a1" containerName="revoke-keys" Oct 30 15:54:22.427826 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.427804 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.463857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.463830 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4jtfs\"" Oct 30 15:54:22.464063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.463830 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 15:54:22.464063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.463837 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 15:54:22.464063 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.463906 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 15:54:22.464238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.463918 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 15:54:22.466335 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.466304 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4"] Oct 30 15:54:22.474529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.474497 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.474689 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.474671 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.474759 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.474740 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.474857 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.474833 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvdbh\" (UniqueName: \"kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.576047 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.576019 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.576181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.576082 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.576181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.576114 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.576181 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.576140 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvdbh\" (UniqueName: \"kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.576694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.576669 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.578483 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.578463 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.578594 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.578577 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.603399 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.603376 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvdbh\" (UniqueName: \"kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh\") pod \"maas-api-5c48877c75-lhpm4\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:22.741922 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:22.741870 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:23.042377 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:23.042338 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4"] Oct 30 15:54:23.044510 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:54:23.044486 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcec705_ae74_4b6d_be57_a536f58dbdd0.slice/crio-f74fc5df6751ca7a132a1b554a90c7a9920692d0cb6506fbafc10ff78171dfed WatchSource:0}: Error finding container f74fc5df6751ca7a132a1b554a90c7a9920692d0cb6506fbafc10ff78171dfed: Status 404 returned error can't find the container with id f74fc5df6751ca7a132a1b554a90c7a9920692d0cb6506fbafc10ff78171dfed Oct 30 15:54:23.045937 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:23.045915 6554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 15:54:23.475968 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:23.475943 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" event={"ID":"cdcec705-ae74-4b6d-be57-a536f58dbdd0","Type":"ContainerStarted","Data":"f74fc5df6751ca7a132a1b554a90c7a9920692d0cb6506fbafc10ff78171dfed"} Oct 30 15:54:24.032045 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:24.031998 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4"] Oct 30 15:54:24.045595 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:24.045562 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-tpwp4"] Oct 30 15:54:24.152312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:24.152279 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68dca1e-b163-4f25-97cb-f57a280dcfc0" path="/var/lib/kubelet/pods/a68dca1e-b163-4f25-97cb-f57a280dcfc0/volumes" Oct 30 15:54:24.481687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:24.481653 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" event={"ID":"cdcec705-ae74-4b6d-be57-a536f58dbdd0","Type":"ContainerStarted","Data":"2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845"} Oct 30 15:54:24.482123 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:24.481815 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:24.534999 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:24.534953 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" podStartSLOduration=2.10632927 podStartE2EDuration="2.534939623s" totalImagesPullingTime="428.610353ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:24.533502158 +0000 UTC m=+3968.492543563" watchObservedRunningTime="2026-10-30 15:54:24.534939623 +0000 UTC m=+3968.493981027" Oct 30 15:54:30.492148 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.492058 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:30.669378 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.669329 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-6r5sb"] Oct 30 15:54:30.673222 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.673198 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.695501 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.695475 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-6r5sb"] Oct 30 15:54:30.743319 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.743262 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01549532-aee9-432d-9357-bf1c459e1a53-metrics-certs\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.743442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.743343 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01549532-aee9-432d-9357-bf1c459e1a53-maas-api-tls\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.743442 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.743411 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01549532-aee9-432d-9357-bf1c459e1a53-openshift-service-ca\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.743529 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.743509 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27rx\" (UniqueName: \"kubernetes.io/projected/01549532-aee9-432d-9357-bf1c459e1a53-kube-api-access-t27rx\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.844164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.844138 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01549532-aee9-432d-9357-bf1c459e1a53-openshift-service-ca\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.844313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.844190 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27rx\" (UniqueName: \"kubernetes.io/projected/01549532-aee9-432d-9357-bf1c459e1a53-kube-api-access-t27rx\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.844313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.844222 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01549532-aee9-432d-9357-bf1c459e1a53-metrics-certs\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.844313 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.844264 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01549532-aee9-432d-9357-bf1c459e1a53-maas-api-tls\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.844842 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.844823 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/01549532-aee9-432d-9357-bf1c459e1a53-openshift-service-ca\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.846780 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.846760 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01549532-aee9-432d-9357-bf1c459e1a53-metrics-certs\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.846887 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.846815 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/01549532-aee9-432d-9357-bf1c459e1a53-maas-api-tls\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.869737 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.869708 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27rx\" (UniqueName: \"kubernetes.io/projected/01549532-aee9-432d-9357-bf1c459e1a53-kube-api-access-t27rx\") pod \"maas-api-595cc648c-6r5sb\" (UID: \"01549532-aee9-432d-9357-bf1c459e1a53\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:30.986743 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:30.986718 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:31.286687 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:31.286665 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-6r5sb"] Oct 30 15:54:31.288852 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:54:31.288826 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01549532_aee9_432d_9357_bf1c459e1a53.slice/crio-285ac064becf4d6539d809a4b3fbc37a99a61304a4a1a12b17d95cf83edd96f7 WatchSource:0}: Error finding container 285ac064becf4d6539d809a4b3fbc37a99a61304a4a1a12b17d95cf83edd96f7: Status 404 returned error can't find the container with id 285ac064becf4d6539d809a4b3fbc37a99a61304a4a1a12b17d95cf83edd96f7 Oct 30 15:54:31.517008 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:31.516973 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" event={"ID":"01549532-aee9-432d-9357-bf1c459e1a53","Type":"ContainerStarted","Data":"285ac064becf4d6539d809a4b3fbc37a99a61304a4a1a12b17d95cf83edd96f7"} Oct 30 15:54:32.525528 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:32.525488 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" event={"ID":"01549532-aee9-432d-9357-bf1c459e1a53","Type":"ContainerStarted","Data":"c05a941664768e94428e2c6e11ac1e6d29211237b3cb3ffb90beb4aa44a2260b"} Oct 30 15:54:32.525943 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:32.525575 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:32.572950 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:32.572885 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" podStartSLOduration=2.287258161 podStartE2EDuration="2.572867263s" totalImagesPullingTime="285.609102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 15:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:54:32.571800035 +0000 UTC m=+3976.530841441" watchObservedRunningTime="2026-10-30 15:54:32.572867263 +0000 UTC m=+3976.531908668" Oct 30 15:54:38.541103 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:38.541067 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-6r5sb" Oct 30 15:54:38.622418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:38.622382 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4"] Oct 30 15:54:38.622701 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:38.622645 6554 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" podUID="cdcec705-ae74-4b6d-be57-a536f58dbdd0" containerName="maas-api" containerID="cri-o://2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845" gracePeriod=30 Oct 30 15:54:39.200773 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:54:39.200734 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:54:39.881761 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.881737 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:39.968825 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.968753 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs\") pod \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " Oct 30 15:54:39.968964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.968872 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh\" (UniqueName: \"kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh\") pod \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " Oct 30 15:54:39.968964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.968912 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls\") pod \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " Oct 30 15:54:39.968964 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.968960 6554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca\") pod \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\" (UID: \"cdcec705-ae74-4b6d-be57-a536f58dbdd0\") " Oct 30 15:54:39.969424 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.969396 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca" pod "cdcec705-ae74-4b6d-be57-a536f58dbdd0" (UID: "cdcec705-ae74-4b6d-be57-a536f58dbdd0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 15:54:39.970947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.970920 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls" pod "cdcec705-ae74-4b6d-be57-a536f58dbdd0" (UID: "cdcec705-ae74-4b6d-be57-a536f58dbdd0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:39.971052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.970948 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs" pod "cdcec705-ae74-4b6d-be57-a536f58dbdd0" (UID: "cdcec705-ae74-4b6d-be57-a536f58dbdd0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 15:54:39.971052 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:39.970999 6554 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh" pod "cdcec705-ae74-4b6d-be57-a536f58dbdd0" (UID: "cdcec705-ae74-4b6d-be57-a536f58dbdd0"). InnerVolumeSpecName "kube-api-access-hvdbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 15:54:40.070653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.070555 6554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvdbh\" (UniqueName: \"kubernetes.io/projected/cdcec705-ae74-4b6d-be57-a536f58dbdd0-kube-api-access-hvdbh\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:54:40.070653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.070585 6554 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-maas-api-tls\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:54:40.070653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.070599 6554 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcec705-ae74-4b6d-be57-a536f58dbdd0-openshift-service-ca\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:54:40.070653 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.070632 6554 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cdcec705-ae74-4b6d-be57-a536f58dbdd0-metrics-certs\") on node \"ip-10-0-139-204.ec2.internal\" DevicePath \"\"" Oct 30 15:54:40.584206 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.584156 6554 generic.go:356] "Generic (PLEG): container finished" podID="cdcec705-ae74-4b6d-be57-a536f58dbdd0" containerID="2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845" exitCode=0 Oct 30 15:54:40.584457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.584235 6554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" Oct 30 15:54:40.584457 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.584280 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" event={"ID":"cdcec705-ae74-4b6d-be57-a536f58dbdd0","Type":"ContainerDied","Data":"2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845"} Oct 30 15:54:40.584606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.584459 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4" event={"ID":"cdcec705-ae74-4b6d-be57-a536f58dbdd0","Type":"ContainerDied","Data":"f74fc5df6751ca7a132a1b554a90c7a9920692d0cb6506fbafc10ff78171dfed"} Oct 30 15:54:40.584606 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.584522 6554 scope.go:122] "RemoveContainer" containerID="2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845" Oct 30 15:54:40.595451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.595426 6554 scope.go:122] "RemoveContainer" containerID="2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845" Oct 30 15:54:40.595789 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:54:40.595762 6554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845\": container with ID starting with 2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845 not found: ID does not exist" containerID="2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845" Oct 30 15:54:40.595924 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.595795 6554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845"} err="failed to get container status \"2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845\": rpc error: code = NotFound desc = could not find container \"2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845\": container with ID starting with 2fc380bfa22193b7128d31072f73f03fda85ccdcf22ba119fa1242ff0729a845 not found: ID does not exist" Oct 30 15:54:40.641959 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.641928 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4"] Oct 30 15:54:40.655585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:40.655561 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lhpm4"] Oct 30 15:54:42.152322 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:54:42.152283 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcec705-ae74-4b6d-be57-a536f58dbdd0" path="/var/lib/kubelet/pods/cdcec705-ae74-4b6d-be57-a536f58dbdd0/volumes" Oct 30 15:55:09.201274 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:55:09.201237 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:55:18.748141 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:18.748105 6554 scope.go:122] "RemoveContainer" containerID="229be308198506373801af497fe06ce280a6b572d1dfacd412ee3ead357ad267" Oct 30 15:55:18.759517 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:18.759490 6554 scope.go:122] "RemoveContainer" containerID="353e195665364eaa4437238f28d4a4807be126188ea38b67a0dc0e35ac2f2edb" Oct 30 15:55:21.002121 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:21.002090 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ff567df79-mrmbk_d0ef798f-d74c-4d9a-8c55-c40ed34976a9/authorino/0.log" Oct 30 15:55:33.330163 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:33.330120 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-r5956_e88e05d5-b66e-4737-b87b-19fd58909603/manager/0.log" Oct 30 15:55:34.032902 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:34.032873 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw_8cc79f79-0d0c-4670-9921-101d339eb468/manager/0.log" Oct 30 15:55:34.061947 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:34.061922 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw_8cc79f79-0d0c-4670-9921-101d339eb468/copy-manifests/0.log" Oct 30 15:55:35.337685 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:35.337650 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-6r5sb_01549532-aee9-432d-9357-bf1c459e1a53/maas-api/0.log" Oct 30 15:55:35.911001 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:35.910978 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-f9zzh_fb78ed19-043e-49f8-aef5-746fd6cd03a1/revoke-keys/0.log" Oct 30 15:55:36.097565 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:36.097540 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-xt5rg_28ba3fd2-8e47-4656-8899-063bbb81b052/revoke-keys/0.log" Oct 30 15:55:36.477801 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:36.477773 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5d054765-wl5sf_c1ea4f16-d76d-4095-846c-7a5e76f8e9af/revoke-keys/0.log" Oct 30 15:55:36.860451 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:36.860388 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-jbr84_38e1d5ea-0e63-4d82-b07b-fb8a772f1114/revoke-keys/0.log" Oct 30 15:55:38.955002 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:38.954967 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ff567df79-mrmbk_d0ef798f-d74c-4d9a-8c55-c40ed34976a9/authorino/0.log" Oct 30 15:55:39.201163 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:55:39.201123 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:55:39.625475 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:39.625454 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-x7gb5_7ff199bb-f4ae-408f-b5c1-3e172f42a81e/kuadrant-console-plugin/0.log" Oct 30 15:55:40.292609 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:40.292574 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-8mhhv_6875de95-ef0d-4239-a549-74086c0d5b0f/manager/0.log" Oct 30 15:55:41.288164 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:41.288137 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vkwdx_3c8367b1-00aa-40d7-ab02-e25379a096c0/discovery/0.log" Oct 30 15:55:41.492635 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:41.492612 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d8f97cc56-pzc99_dad6de2a-89da-437a-9267-28357e2f8c77/kube-auth-proxy/0.log" Oct 30 15:55:42.097812 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:42.097782 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9drkm_7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f/istio-proxy/0.log" Oct 30 15:55:42.699202 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:42.699170 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jdv8h_5faca140-9640-4e89-9dc2-77479c5e0f64/istio-proxy/0.log" Oct 30 15:55:43.323738 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:43.323704 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-twvbm_2775a872-b309-41d5-a34c-cf822339e896/istio-proxy/0.log" Oct 30 15:55:45.505208 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:45.505173 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8rxfm_caa663cd-eb7c-4b24-8601-4a9256fd097c/main/0.log" Oct 30 15:55:45.907197 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:45.907165 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-pspnq_ae429169-b691-4914-8603-c992a490012c/main/0.log" Oct 30 15:55:46.317367 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:46.317277 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn_2573e9d0-9c8d-4dfa-8336-0b0804094d4f/main/0.log" Oct 30 15:55:46.524149 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:46.524122 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh_72284b4e-d5b0-4317-b86f-7341f3d3364f/main/0.log" Oct 30 15:55:47.417419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:47.417395 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ff567df79-mrmbk_d0ef798f-d74c-4d9a-8c55-c40ed34976a9/authorino/0.log" Oct 30 15:55:58.242811 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:58.242727 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-r5956_e88e05d5-b66e-4737-b87b-19fd58909603/manager/0.log" Oct 30 15:55:59.001027 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:59.001001 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw_8cc79f79-0d0c-4670-9921-101d339eb468/copy-manifests/0.log" Oct 30 15:55:59.030336 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:55:59.030312 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f6f56bfdb-6wnbw_8cc79f79-0d0c-4670-9921-101d339eb468/manager/0.log" Oct 30 15:56:00.076894 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:00.076860 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-6r5sb_01549532-aee9-432d-9357-bf1c459e1a53/maas-api/0.log" Oct 30 15:56:00.670799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:00.670767 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-f9zzh_fb78ed19-043e-49f8-aef5-746fd6cd03a1/revoke-keys/0.log" Oct 30 15:56:00.848285 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:00.848250 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-xt5rg_28ba3fd2-8e47-4656-8899-063bbb81b052/revoke-keys/0.log" Oct 30 15:56:01.199596 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:01.199568 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5d054765-wl5sf_c1ea4f16-d76d-4095-846c-7a5e76f8e9af/revoke-keys/0.log" Oct 30 15:56:01.557033 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:01.556966 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-jbr84_38e1d5ea-0e63-4d82-b07b-fb8a772f1114/revoke-keys/0.log" Oct 30 15:56:03.358725 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:03.358691 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ff567df79-mrmbk_d0ef798f-d74c-4d9a-8c55-c40ed34976a9/authorino/0.log" Oct 30 15:56:04.008175 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:04.008151 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-x7gb5_7ff199bb-f4ae-408f-b5c1-3e172f42a81e/kuadrant-console-plugin/0.log" Oct 30 15:56:04.668418 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:04.668389 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-8mhhv_6875de95-ef0d-4239-a549-74086c0d5b0f/manager/0.log" Oct 30 15:56:05.472871 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:05.472840 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vkwdx_3c8367b1-00aa-40d7-ab02-e25379a096c0/discovery/0.log" Oct 30 15:56:05.668763 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:05.668733 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d8f97cc56-pzc99_dad6de2a-89da-437a-9267-28357e2f8c77/kube-auth-proxy/0.log" Oct 30 15:56:06.251768 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:06.251740 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9drkm_7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f/istio-proxy/0.log" Oct 30 15:56:06.838852 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:06.838825 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jdv8h_5faca140-9640-4e89-9dc2-77479c5e0f64/istio-proxy/0.log" Oct 30 15:56:07.438585 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:07.438556 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-twvbm_2775a872-b309-41d5-a34c-cf822339e896/istio-proxy/0.log" Oct 30 15:56:09.200876 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:56:09.200841 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:56:09.231120 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:09.231096 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-8rxfm_caa663cd-eb7c-4b24-8601-4a9256fd097c/main/0.log" Oct 30 15:56:09.644522 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:09.644501 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-pspnq_ae429169-b691-4914-8603-c992a490012c/main/0.log" Oct 30 15:56:10.056445 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:10.056383 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-8xnzn_2573e9d0-9c8d-4dfa-8336-0b0804094d4f/main/0.log" Oct 30 15:56:10.254112 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:10.254086 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-z2tkh_72284b4e-d5b0-4317-b86f-7341f3d3364f/main/0.log" Oct 30 15:56:18.787000 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:18.786968 6554 scope.go:122] "RemoveContainer" containerID="6a048cb2d19ad7d4a6ae26a7a536d2fdf1227a1f4b299aad7c0c1e60d78ab725" Oct 30 15:56:32.030670 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:32.030639 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84"] Oct 30 15:56:32.044059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:32.044033 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-jbr84"] Oct 30 15:56:32.153059 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:56:32.153022 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e1d5ea-0e63-4d82-b07b-fb8a772f1114" path="/var/lib/kubelet/pods/38e1d5ea-0e63-4d82-b07b-fb8a772f1114/volumes" Oct 30 15:56:39.202279 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:56:39.202236 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:57:09.200227 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:57:09.200185 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:57:22.046824 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:22.046785 6554 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf"] Oct 30 15:57:22.060278 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:22.060249 6554 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d054765-wl5sf"] Oct 30 15:57:22.152433 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:22.152403 6554 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1ea4f16-d76d-4095-846c-7a5e76f8e9af" path="/var/lib/kubelet/pods/c1ea4f16-d76d-4095-846c-7a5e76f8e9af/volumes" Oct 30 15:57:36.203179 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:36.203101 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bz4fj_1c36de38-25e2-4eb1-9b01-a7c342084792/global-pull-secret-syncer/0.log" Oct 30 15:57:37.553062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:37.553029 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sbcnp_8e5f527d-4282-4ace-b004-5a2504a1d73e/konnectivity-agent/0.log" Oct 30 15:57:37.836423 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:37.836371 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-204.ec2.internal_c47c2ff730e7510466e8747f4e5f45ca/haproxy/0.log" Oct 30 15:57:39.200969 ip-10-0-139-204 kubenswrapper[6554]: E1030 15:57:39.200927 6554 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hj6kg" containerName="extractor" Oct 30 15:57:43.853071 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:43.853041 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7ff567df79-mrmbk_d0ef798f-d74c-4d9a-8c55-c40ed34976a9/authorino/0.log" Oct 30 15:57:44.255799 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:44.255774 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-x7gb5_7ff199bb-f4ae-408f-b5c1-3e172f42a81e/kuadrant-console-plugin/0.log" Oct 30 15:57:44.712790 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:44.712766 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-8mhhv_6875de95-ef0d-4239-a549-74086c0d5b0f/manager/0.log" Oct 30 15:57:46.214321 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.214283 6554 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb"] Oct 30 15:57:46.214766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.214702 6554 cpu_manager.go:414] "containerMap: removing container" podUID="cdcec705-ae74-4b6d-be57-a536f58dbdd0" containerName="maas-api" Oct 30 15:57:46.214766 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.214716 6554 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcec705-ae74-4b6d-be57-a536f58dbdd0" containerName="maas-api" Oct 30 15:57:46.214855 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.214796 6554 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcec705-ae74-4b6d-be57-a536f58dbdd0" containerName="maas-api" Oct 30 15:57:46.218043 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.218020 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.239219 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.239189 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb"] Oct 30 15:57:46.251169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.251149 6554 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cbkvm\"/\"default-dockercfg-wlh7f\"" Oct 30 15:57:46.251169 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.251158 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbkvm\"/\"openshift-service-ca.crt\"" Oct 30 15:57:46.251422 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.251174 6554 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cbkvm\"/\"kube-root-ca.crt\"" Oct 30 15:57:46.340874 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.340851 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-proc\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.340982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.340884 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-sys\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.340982 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.340914 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq4c5\" (UniqueName: \"kubernetes.io/projected/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-kube-api-access-xq4c5\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.341062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.341001 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-lib-modules\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.341062 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.341019 6554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-podres\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442124 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442101 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-sys\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442141 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq4c5\" (UniqueName: \"kubernetes.io/projected/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-kube-api-access-xq4c5\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442232 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442220 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-sys\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442256 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-lib-modules\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442312 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442283 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-podres\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442419 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442381 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-lib-modules\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442431 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-podres\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442462 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442433 6554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-proc\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.442530 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.442467 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-proc\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.467203 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.467144 6554 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq4c5\" (UniqueName: \"kubernetes.io/projected/2af06afb-8d7e-4cc2-b5b1-003c59ae878f-kube-api-access-xq4c5\") pod \"perf-node-gather-daemonset-blrpb\" (UID: \"2af06afb-8d7e-4cc2-b5b1-003c59ae878f\") " pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.531525 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.531507 6554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:46.828553 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:46.828528 6554 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb"] Oct 30 15:57:46.830985 ip-10-0-139-204 kubenswrapper[6554]: W1030 15:57:46.830951 6554 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2af06afb_8d7e_4cc2_b5b1_003c59ae878f.slice/crio-7b2392db6a2144c62b61a5fbfac537cc7238e7669baec211a908c3e02ac4d2dd WatchSource:0}: Error finding container 7b2392db6a2144c62b61a5fbfac537cc7238e7669baec211a908c3e02ac4d2dd: Status 404 returned error can't find the container with id 7b2392db6a2144c62b61a5fbfac537cc7238e7669baec211a908c3e02ac4d2dd Oct 30 15:57:47.217256 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.217218 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9r8cw_dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/kube-state-metrics/0.log" Oct 30 15:57:47.332331 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.332306 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9r8cw_dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/kube-rbac-proxy-main/0.log" Oct 30 15:57:47.453785 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.453751 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-9r8cw_dcea1b1f-2eb6-466b-b72f-b1e7d6e50117/kube-rbac-proxy-self/0.log" Oct 30 15:57:47.544677 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.544593 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" event={"ID":"2af06afb-8d7e-4cc2-b5b1-003c59ae878f","Type":"ContainerStarted","Data":"970f2133a2f7b1ff15728b9eba54968333ed2c3e2255451084a2b2710f70f9de"} Oct 30 15:57:47.544827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.544689 6554 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" event={"ID":"2af06afb-8d7e-4cc2-b5b1-003c59ae878f","Type":"ContainerStarted","Data":"7b2392db6a2144c62b61a5fbfac537cc7238e7669baec211a908c3e02ac4d2dd"} Oct 30 15:57:47.544827 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.544785 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:47.579250 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.579221 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cbbd6bfb5-h9xsq_525049c6-49f7-46f1-8f0e-b31091e03f26/metrics-server/0.log" Oct 30 15:57:47.589238 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.589188 6554 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" podStartSLOduration=1.5891727580000001 podStartE2EDuration="1.589172758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 15:57:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 15:57:47.588097864 +0000 UTC m=+4171.547139266" watchObservedRunningTime="2026-10-30 15:57:47.589172758 +0000 UTC m=+4171.548214163" Oct 30 15:57:47.841292 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.841213 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jkbnv_e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/node-exporter/0.log" Oct 30 15:57:47.965096 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:47.965071 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jkbnv_e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/kube-rbac-proxy/0.log" Oct 30 15:57:48.087547 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:48.087525 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jkbnv_e7e2a6a8-11f0-49bd-a0ac-1bcf094bd076/init-textfile/0.log" Oct 30 15:57:49.010240 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.010212 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8829w_0a396a10-b996-4d83-9783-f4a6a543ff6e/kube-rbac-proxy-main/0.log" Oct 30 15:57:49.131415 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.131389 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8829w_0a396a10-b996-4d83-9783-f4a6a543ff6e/kube-rbac-proxy-self/0.log" Oct 30 15:57:49.254042 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.254012 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-8829w_0a396a10-b996-4d83-9783-f4a6a543ff6e/openshift-state-metrics/0.log" Oct 30 15:57:49.385690 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.385630 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/prometheus/0.log" Oct 30 15:57:49.506277 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.506247 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/config-reloader/0.log" Oct 30 15:57:49.626589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.626561 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/thanos-sidecar/0.log" Oct 30 15:57:49.747325 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.747304 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/kube-rbac-proxy-web/0.log" Oct 30 15:57:49.891212 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:49.891190 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/kube-rbac-proxy/0.log" Oct 30 15:57:50.027723 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:50.027669 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/kube-rbac-proxy-thanos/0.log" Oct 30 15:57:50.154694 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:50.154676 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b6cf0c34-2548-40ca-81df-b06ca85f0055/init-config-reloader/0.log" Oct 30 15:57:50.692607 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:50.692581 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d6ccfcd7f-rs5lq_5e0152b3-9c26-4191-bfaf-182715417b9b/telemeter-client/0.log" Oct 30 15:57:50.813715 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:50.813698 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d6ccfcd7f-rs5lq_5e0152b3-9c26-4191-bfaf-182715417b9b/reload/0.log" Oct 30 15:57:50.935537 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:50.935506 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5d6ccfcd7f-rs5lq_5e0152b3-9c26-4191-bfaf-182715417b9b/kube-rbac-proxy/0.log" Oct 30 15:57:53.562115 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:53.562083 6554 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cbkvm/perf-node-gather-daemonset-blrpb" Oct 30 15:57:55.498906 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:55.498873 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5dcc966694-t2bvg_c6858670-0950-46e8-8db9-06e2f3b4aa24/console/0.log" Oct 30 15:57:59.839589 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:59.839558 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmw9l_13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/dns/0.log" Oct 30 15:57:59.959712 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:57:59.959687 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmw9l_13d07af5-d2a4-401e-9d97-9b98c4f6e2d8/kube-rbac-proxy/0.log" Oct 30 15:58:00.216258 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:00.216232 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5b49_da158b42-504b-4dfa-9b41-3333f4bbf0e7/dns-node-resolver/0.log" Oct 30 15:58:01.470077 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:01.470037 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-vkwdx_3c8367b1-00aa-40d7-ab02-e25379a096c0/discovery/0.log" Oct 30 15:58:01.503465 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:01.503441 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d8f97cc56-pzc99_dad6de2a-89da-437a-9267-28357e2f8c77/kube-auth-proxy/0.log" Oct 30 15:58:01.749234 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:01.749177 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-9drkm_7c1a9bc0-b6f3-48c5-888e-0d44cd70e74f/istio-proxy/0.log" Oct 30 15:58:01.929127 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:01.929102 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-jdv8h_5faca140-9640-4e89-9dc2-77479c5e0f64/istio-proxy/0.log" Oct 30 15:58:01.937660 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:01.937639 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vvpcd_5a72a542-4dd5-46f1-bdba-e82aba26d948/node-ca/0.log" Oct 30 15:58:02.146116 ip-10-0-139-204 kubenswrapper[6554]: I1030 15:58:02.146094 6554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-twvbm_2775a872-b309-41d5-a34c-cf822339e896/istio-proxy/0.log"